[jira] [Resolved] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps
[ https://issues.apache.org/jira/browse/IMPALA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pooja Nilangekar resolved IMPALA-8245. -- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Add hostname to timeout error message in HdfsMonitoredOps > - > > Key: IMPALA-8245 > URL: https://issues.apache.org/jira/browse/IMPALA-8245 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Pooja Nilangekar >Priority: Major > Fix For: Impala 3.2.0 > > > If a DiskIo operation times out, it generates a > TErrorCode::THREAD_POOL_TASK_TIMED_OUT or > TErrorCode::THREAD_POOL_SUBMIT_FAILED error codes. These call > GetDescription() to get DiskIo related context. That information should > include the hostname where the error occurred to allow tracking down a > problematic host that is seeing DiskIo timeouts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps
[ https://issues.apache.org/jira/browse/IMPALA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pooja Nilangekar resolved IMPALA-8245. -- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Add hostname to timeout error message in HdfsMonitoredOps > - > > Key: IMPALA-8245 > URL: https://issues.apache.org/jira/browse/IMPALA-8245 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Pooja Nilangekar >Priority: Major > Fix For: Impala 3.2.0 > > > If a DiskIo operation times out, it generates a > TErrorCode::THREAD_POOL_TASK_TIMED_OUT or > TErrorCode::THREAD_POOL_SUBMIT_FAILED error codes. These call > GetDescription() to get DiskIo related context. That information should > include the hostname where the error occurred to allow tracking down a > problematic host that is seeing DiskIo timeouts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749308#comment-16749308
]
Michael Ho edited comment on IMPALA-8027 at 2/27/19 4:20 AM:
-
The profile didn't suggest much either. The last report time for the
coordinator fragment was 05:16:55, which was when the profile was initially
created. So, the coordinator itself didn't receive any update from the
coordinator fragment itself for two minutes. This suggests the Prepare() didn't
complete within the time frame.
{noformat}
Instance 194f5b70907ac97c:84a116d6
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000)
Last report received time: 2018-12-28 05:16:52.214
{noformat}
FWIW, other fragment instances running on other Impalad peers were able to send
report so it seems the status reporting handling was working on coordinator
({{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}):
{noformat}
Instance 194f5b70907ac97c:84a116d60003
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001)
Last report received time: 2018-12-28 05:18:56.204
Hdfs split stats (:<# splits>/): -1:4/152.76 KB
{noformat}
The fragment instances running on
{{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}
eventually completed.
{noformat}
I1228 05:16:52.217275 26685 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d6 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=7
I1228 05:20:08.981983 26685 krpc-data-stream-mgr.cc:294] DeregisterRecvr():
fragment_instance_id=194f5b70907ac97c:84a116d6, node=2
I1228 05:20:08.982043 26685 query-state.cc:576] Instance completed.
instance_id=194f5b70907ac97c:84a116d6 #in-flight=5 status=CANCELLED:
Cancelled
{noformat}
{noformat}
I1228 05:16:52.217344 26686 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d60001 fragment_idx=1
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=8
I1228 05:20:08.992434 26686 llvm-codegen.cc:1104] For query
194f5b70907ac97c:84a116d6 the following functions were not finalized
and have been removed from the module:
I1228 05:20:09.049794 26686 query-state.cc:576] Instance completed.
instance_id=194f5b70907ac97c:84a116d60001 #in-flight=5 status=CANCELLED:
Cancelled
I1228 05:20:09.056677 26686 query-exec-mgr.cc:182] ReleaseQueryState(): deleted
query_id=194f5b70907ac97c:84a116d6
{noformat}
With the data available so far, one could only conclude that the test failed
due to some flaky scheduling issue which slowed down some query fragments
running on
{{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}. Given
this hasn't happened again since the initial reporting, I don't think there is
much we need to do at this point.
Longer term, fixing IMPALA-6818 will remove this class of flaky timeout issues.
was (Author: kwho):
The profile didn't suggest much either. The last report time for the
coordinator fragment was 05:16:55, which was when the profile was initially
created. So, the coordinator itself didn't receive any update from the
coordinator fragment itself for two minutes. This suggests the Prepare() didn't
complete within the time frame.
{noformat}
Instance 194f5b70907ac97c:84a116d6
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000)
Last report received time: 2018-12-28 05:16:52.214
{noformat}
FWIW, other fragment instances running on other Impalad peers were able to send
report so it seems the status reporting handling was working on coordinator
({{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}):
{noformat}
Instance 194f5b70907ac97c:84a116d60003
(host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001)
Last report received time: 2018-12-28 05:18:56.204
Hdfs split stats (:<# splits>/): -1:4/152.76 KB
{noformat}
The fragment instances running on
{{impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000}}
eventually completed.
{noformat}
I1228 05:16:52.217275 26685 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d6 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=7
I1228 05:20:08.981983 26685 krpc-data-stream-mgr.cc:294] DeregisterRecvr():
fragment_instance_id=194f5b70907ac97c:84a116d6, node=2
I1228 05:20:08.982043 26685 query-state.cc:576] Instance completed.
instance_id=194f5b70907ac97c:84a116d6 #in-flight=5 status=CANCELLED:
Cancelled
{noformat}
{noformat}
I1228 05:16:52.217344 26686 query-state.cc:568] Executing instance.
instance_id=194f5b70907ac97c:84a116d60001 fragment_idx=1
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=8
I1228 05:20:08.992434 26686 llvm-codegen.cc:1104] For query
194f5b70907ac97c:84a116d6 the
[jira] [Comment Edited] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778866#comment-16778866
]
Michael Ho edited comment on IMPALA-8027 at 2/27/19 4:19 AM:
-
Haven't happened in a while. The long term fix is IMPALA-6818
was (Author: kwho):
Haven't happened in a while. The long term fix is IMPALA-6618.
> KRPC datastream timing out on both the receiver and sender side even in a
> minicluster
> -
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Bikramjeet Vig
>Assignee: Michael Ho
>Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and
> receiver causing two running queries to fail. This happened while running
> core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id
> 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID
> cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377 coordinator.cc:707] Backend completed:
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> remaining=2 query_id=8f46b2518734bef1:6ef2d404
> I1228 05:18:56.204756 13377 coordinator-backend-state.cc:262]
> query_id=8f46b2518734bef1:6ef2d404: first in-progress backend:
> impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000
> I1228 05:18:56.204769 13377 coordinator.cc:522] ExecState: query
> id=8f46b2518734bef1:6ef2d404
> finstance=8f46b2518734bef1:6ef2d4040001 on
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> (EXECUTING -> ERROR) status=Sender 127.0.0.1 timed out waiting for receiver
> fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> {noformat}
> Logs from executor:
> {noformat}
> E1228 05:18:56.203181 26715 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=8f46b2518734bef1:6ef2d404): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> E1228 05:18:56.203256 26682 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=194f5b70907ac97c:84a116d6): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203451 26715 query-state.cc:576] Instance completed.
> instance_id=8f46b2518734bef1:6ef2d4040001 #in-flight=3
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203485 26713 query-state.cc:249] UpdateBackendExecState():
> last report for 8f46b2518734bef1:6ef2d404
> I1228 05:18:56.203514 26682 query-state.cc:576] Instance completed.
> instance_id=194f5b70907ac97c:84a116d60003 #in-flight=2
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203536 26680 query-state.cc:249] UpdateBackendExecState():
> last report for 194f5b70907ac97c:84a116d6
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8027.
Resolution: Duplicate
Haven't happened in a while. The long term fix is IMPALA-6618.
> KRPC datastream timing out on both the receiver and sender side even in a
> minicluster
> -
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Bikramjeet Vig
>Assignee: Michael Ho
>Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and
> receiver causing two running queries to fail. This happened while running
> core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id
> 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID
> cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377 coordinator.cc:707] Backend completed:
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> remaining=2 query_id=8f46b2518734bef1:6ef2d404
> I1228 05:18:56.204756 13377 coordinator-backend-state.cc:262]
> query_id=8f46b2518734bef1:6ef2d404: first in-progress backend:
> impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000
> I1228 05:18:56.204769 13377 coordinator.cc:522] ExecState: query
> id=8f46b2518734bef1:6ef2d404
> finstance=8f46b2518734bef1:6ef2d4040001 on
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> (EXECUTING -> ERROR) status=Sender 127.0.0.1 timed out waiting for receiver
> fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> {noformat}
> Logs from executor:
> {noformat}
> E1228 05:18:56.203181 26715 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=8f46b2518734bef1:6ef2d404): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> E1228 05:18:56.203256 26682 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=194f5b70907ac97c:84a116d6): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203451 26715 query-state.cc:576] Instance completed.
> instance_id=8f46b2518734bef1:6ef2d4040001 #in-flight=3
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203485 26713 query-state.cc:249] UpdateBackendExecState():
> last report for 8f46b2518734bef1:6ef2d404
> I1228 05:18:56.203514 26682 query-state.cc:576] Instance completed.
> instance_id=194f5b70907ac97c:84a116d60003 #in-flight=2
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203536 26680 query-state.cc:249] UpdateBackendExecState():
> last report for 194f5b70907ac97c:84a116d6
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8027) KRPC datastream timing out on both the receiver and sender side even in a minicluster
[
https://issues.apache.org/jira/browse/IMPALA-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8027.
Resolution: Duplicate
Haven't happened in a while. The long term fix is IMPALA-6618.
> KRPC datastream timing out on both the receiver and sender side even in a
> minicluster
> -
>
> Key: IMPALA-8027
> URL: https://issues.apache.org/jira/browse/IMPALA-8027
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Bikramjeet Vig
>Assignee: Michael Ho
>Priority: Critical
> Labels: broken-build
>
> krpc datastreams seem to time out at the same time at both sender and
> receiver causing two running queries to fail. This happened while running
> core tests on s3.
> Logs from coordinator:
> {noformat}
> I1228 05:18:56.202587 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203061 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11274) took 120782ms. Request Metrics: {}
> I1228 05:18:56.203114 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203136 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53110 (request call id
> 8637) took 123811ms. Request Metrics: {}
> I1228 05:18:56.203155 13396 krpc-data-stream-mgr.cc:353] Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203167 13396 rpcz_store.cc:265] Call
> impala.DataStreamService.TransmitData from 127.0.0.1:53118 (request call id
> 11273) took 123776ms. Request Metrics: {}
> I1228 05:18:56.203181 13396 krpc-data-stream-mgr.cc:408] Reduced stream ID
> cache from 413 items, to 410, eviction took: 1ms
> I1228 05:18:56.204746 13377 coordinator.cc:707] Backend completed:
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> remaining=2 query_id=8f46b2518734bef1:6ef2d404
> I1228 05:18:56.204756 13377 coordinator-backend-state.cc:262]
> query_id=8f46b2518734bef1:6ef2d404: first in-progress backend:
> impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22000
> I1228 05:18:56.204769 13377 coordinator.cc:522] ExecState: query
> id=8f46b2518734bef1:6ef2d404
> finstance=8f46b2518734bef1:6ef2d4040001 on
> host=impala-ec2-centos74-m5-4xlarge-ondemand-07b3.vpc.cloudera.com:22001
> (EXECUTING -> ERROR) status=Sender 127.0.0.1 timed out waiting for receiver
> fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> {noformat}
> Logs from executor:
> {noformat}
> E1228 05:18:56.203181 26715 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=8f46b2518734bef1:6ef2d404): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 8f46b2518734bef1:6ef2d404, dest node: 2
> E1228 05:18:56.203256 26682 krpc-data-stream-sender.cc:343] channel send to
> 127.0.0.1:27000 failed:
> (fragment_instance_id=194f5b70907ac97c:84a116d6): Sender 127.0.0.1
> timed out waiting for receiver fragment instance:
> 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203451 26715 query-state.cc:576] Instance completed.
> instance_id=8f46b2518734bef1:6ef2d4040001 #in-flight=3
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 8f46b2518734bef1:6ef2d404, dest node: 2
> I1228 05:18:56.203485 26713 query-state.cc:249] UpdateBackendExecState():
> last report for 8f46b2518734bef1:6ef2d404
> I1228 05:18:56.203514 26682 query-state.cc:576] Instance completed.
> instance_id=194f5b70907ac97c:84a116d60003 #in-flight=2
> status=DATASTREAM_SENDER_TIMEOUT: Sender 127.0.0.1 timed out waiting for
> receiver fragment instance: 194f5b70907ac97c:84a116d6, dest node: 2
> I1228 05:18:56.203536 26680 query-state.cc:249] UpdateBackendExecState():
> last report for 194f5b70907ac97c:84a116d6
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
[
https://issues.apache.org/jira/browse/IMPALA-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8251.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Impala fails to start for
> TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
> --
>
> Key: IMPALA-8251
> URL: https://issues.apache.org/jira/browse/IMPALA-8251
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.2.0
>
>
> custom_cluster/test_exchange_deferred_batches.py is hitting the following
> error on release builds:
> {noformat}
> ERROR at setup of
> TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> common/custom_cluster_test_suite.py:155: in setup_method
> expected_num_executors=cluster_size)
> common/custom_cluster_test_suite.py:230: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> /usr/lib/python2.7/subprocess.py:541: in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
> '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
> '--impalad_args=--stress_datastream_recvr_delay_ms=3000
> --exchg_node_buffer_size_bytes=1024
> --datastream_service_num_deserialization_threads=1 ',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50 ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> The impalad logs show errors:
> {noformat}
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address
> for: jhzvlthd
> . Impalad exiting.
> {noformat}
> It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
[
https://issues.apache.org/jira/browse/IMPALA-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8251.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Impala fails to start for
> TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
> --
>
> Key: IMPALA-8251
> URL: https://issues.apache.org/jira/browse/IMPALA-8251
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.2.0
>
>
> custom_cluster/test_exchange_deferred_batches.py is hitting the following
> error on release builds:
> {noformat}
> ERROR at setup of
> TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> common/custom_cluster_test_suite.py:155: in setup_method
> expected_num_executors=cluster_size)
> common/custom_cluster_test_suite.py:230: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> /usr/lib/python2.7/subprocess.py:541: in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
> '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
> '--impalad_args=--stress_datastream_recvr_delay_ms=3000
> --exchg_node_buffer_size_bytes=1024
> --datastream_service_num_deserialization_threads=1 ',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50 ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> The impalad logs show errors:
> {noformat}
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address
> for: jhzvlthd
> . Impalad exiting.
> {noformat}
> It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
[
https://issues.apache.org/jira/browse/IMPALA-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778864#comment-16778864
]
ASF subversion and git services commented on IMPALA-8251:
-
Commit ca89014eaabdb8a74f1ff7643f65020a03ce82ab in impala's branch
refs/heads/master from Michael Ho
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ca89014 ]
IMPALA-8251: Run test_exchange_deferred_batches in dev builds only
The startup flag stress_datastream_recvr_delay_ms is not available in
development builds. Skip the test in non-developement builds.
Testing done: Ran the test with release build and verified that it's skipped.
Change-Id: I5caaa6fa39d6c97f313b675838c27740af9aa1d5
Reviewed-on: http://gerrit.cloudera.org:8080/12610
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Impala fails to start for
> TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
> --
>
> Key: IMPALA-8251
> URL: https://issues.apache.org/jira/browse/IMPALA-8251
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
>
> custom_cluster/test_exchange_deferred_batches.py is hitting the following
> error on release builds:
> {noformat}
> ERROR at setup of
> TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> common/custom_cluster_test_suite.py:155: in setup_method
> expected_num_executors=cluster_size)
> common/custom_cluster_test_suite.py:230: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> /usr/lib/python2.7/subprocess.py:541: in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
> '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
> '--impalad_args=--stress_datastream_recvr_delay_ms=3000
> --exchg_node_buffer_size_bytes=1024
> --datastream_service_num_deserialization_threads=1 ',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50 ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> The impalad logs show errors:
> {noformat}
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address
> for: jhzvlthd
> . Impalad exiting.
> {noformat}
> It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8250) Impala crashes with -Xcheck:jni
[ https://issues.apache.org/jira/browse/IMPALA-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778865#comment-16778865 ] ASF subversion and git services commented on IMPALA-8250: - Commit b74d63222875482c7efe38b8b2e5ff091a94fb6a in impala's branch refs/heads/master from Philip Zeyliger [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b74d632 ] IMPALA-8250: Fix a handful of JNI usage errors. Running Impala with LIB_HDFS_OPTS=-Xcheck:jni revealed a case of "FATAL ERROR in native method: Bad global or local ref passed to JNI" as well as 'WARNING in native method: JNI FindClass received a bad class descriptor "Lorg/apache/impala/common/Pair;"' This commit fixes these. There are additional JNI warnings that I'm working through as well as a fundamental question about how to avoid these systematically. Kudos to Lars Volker and Sahil Takiar who co-found these issues. Change-Id: Id3b8bccee20cc882cffa4439e550e1f48347d3a6 Reviewed-on: http://gerrit.cloudera.org:8080/12582 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Impala crashes with -Xcheck:jni > --- > > Key: IMPALA-8250 > URL: https://issues.apache.org/jira/browse/IMPALA-8250 > Project: IMPALA > Issue Type: Task >Reporter: Philip Zeyliger >Priority: Major > > The JVM has a checker for JNI usage, and Impala (and libhdfs) have some > violations. This ticket captures figuring that out. At least one of the > issues can crash Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8188) Some SSDs are not properly detected as non-rotational
[
https://issues.apache.org/jira/browse/IMPALA-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8188 started by Joe McDonnell.
-
> Some SSDs are not properly detected as non-rotational
> -
>
> Key: IMPALA-8188
> URL: https://issues.apache.org/jira/browse/IMPALA-8188
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Critical
>
> Here is an example Impala log:
>
> {noformat}
> I0211 10:50:40.650727 18344 init.cc:288] Disk Info:
> Num disks 2:
> nvme0n (rotational=true)
> nvme0n1p (rotational=true){noformat}
> I logged into an equivalent machine, and the OS sees these as not rotational:
>
>
> {noformat}
> # cat /sys/block/nvme0n1/queue/rotational
> 0
> {noformat}
> Device names that end in a number get trimmed (i.e. /dev/sda2 becomes
> /dev/sda). See
> [https://github.com/apache/impala/blob/master/be/src/util/disk-info.cc#L73-L74]
> These devices don't follow that pattern, so we don't find the right files.
> Neither /sys/block/nvme0n nor /sys/block/nvme0n1p exist, so both fall back to
> being rotational.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running
[ https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8247. --- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are > not running > - > > Key: IMPALA-8247 > URL: https://issues.apache.org/jira/browse/IMPALA-8247 > Project: IMPALA > Issue Type: Bug > Components: Backend, Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Blocker > Labels: broken-build > Fix For: Impala 3.2.0 > > > Several tests in be/src/util were converted to the unified backend test > executable in IMPALA-8071. Unfortunately, some tests stopped running as part > of this change. Specifically, the tests in bit-stream-utils-test.cc and > system-state-info-test.cc were not included in the test binary and stopped > running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files > were not included in the UtilTests library. > These two files should be included in the test executable and we need to > update the validation of the unified test executable > (bin/validate-unified-backend-test-filters.py) to catch this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running
[ https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8247. --- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are > not running > - > > Key: IMPALA-8247 > URL: https://issues.apache.org/jira/browse/IMPALA-8247 > Project: IMPALA > Issue Type: Bug > Components: Backend, Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Blocker > Labels: broken-build > Fix For: Impala 3.2.0 > > > Several tests in be/src/util were converted to the unified backend test > executable in IMPALA-8071. Unfortunately, some tests stopped running as part > of this change. Specifically, the tests in bit-stream-utils-test.cc and > system-state-info-test.cc were not included in the test binary and stopped > running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files > were not included in the UtilTests library. > These two files should be included in the test executable and we need to > update the validation of the unified test executable > (bin/validate-unified-backend-test-filters.py) to catch this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7482) Deadlock with unknown lock holder in JVM in java.security.Provider.getService()
[
https://issues.apache.org/jira/browse/IMPALA-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778744#comment-16778744
]
Todd Lipcon commented on IMPALA-7482:
-
Indeed looks like that JDK bug -- was just coming to exactly the same thought.
We've seen this as early as JDK 1.8.0_102.
> Deadlock with unknown lock holder in JVM in
> java.security.Provider.getService()
> ---
>
> Key: IMPALA-7482
> URL: https://issues.apache.org/jira/browse/IMPALA-7482
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: hang
> Attachments: vb1220-jstack2.out
>
>
> We've seen several instances of these mystery deadlocks in impalad's embedded
> JVM. The signature is a deadlock stemming from sun.security.provider.Sun
> being locked by an unknown owner.
> {noformat}
> Found one Java-level deadlock:
> =
> "Thread-24":
> waiting to lock monitor 0x12364688 (object 0x8027ef30, a
> sun.security.provider.Sun),
> which is held by UNKNOWN_owner_addr=0x14120800
> {noformat}
> If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck.
> If it happens in the Kudu client it also causes hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8133) Impala Doc: Review the list of Known Issues and update for 3.2
[ https://issues.apache.org/jira/browse/IMPALA-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8133 started by Alex Rodoni. --- > Impala Doc: Review the list of Known Issues and update for 3.2 > -- > > Key: IMPALA-8133 > URL: https://issues.apache.org/jira/browse/IMPALA-8133 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_32 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8252) Impala writes malformed thrift profiles
Lars Volker created IMPALA-8252:
---
Summary: Impala writes malformed thrift profiles
Key: IMPALA-8252
URL: https://issues.apache.org/jira/browse/IMPALA-8252
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Lars Volker
Assignee: Lars Volker
The change for IMPALA-1048 writes TRuntimeProfileNode.meta_data
unconditionally, even when both its fields are unset. This trips up the Java
reader code, which expects to find exactly one type of a union to be set. The
resulting error looks like this:
{noformat}
Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized type 0
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:144)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at
com.cloudera.impala.thrift.TRuntimeProfileNodeMetadata.standardSchemeReadValue(TRuntimeProfileNodeMetadata.java:163)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:224)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:213)
at org.apache.thrift.TUnion.read(TUnion.java:138)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1532)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1341)
at
com.cloudera.impala.thrift.TRuntimeProfileNode.read(TRuntimeProfileNode.java:1187)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:426)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:405)
at
com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:339)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at com.cloudera.ipe.util.ThriftUtil.read(ThriftUtil.java:65)
{noformat}
(This particular implementation rewrites the Java namespace for compatibility
reasons, but the code used the latest master commit's thrift files.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IMPALA-7482) Deadlock with unknown lock holder in JVM in java.security.Provider.getService()
[
https://issues.apache.org/jira/browse/IMPALA-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778741#comment-16778741
]
Anthony Baldocchi commented on IMPALA-7482:
---
This looks a lot like
[JDK-8215355|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8215355]
> Deadlock with unknown lock holder in JVM in
> java.security.Provider.getService()
> ---
>
> Key: IMPALA-7482
> URL: https://issues.apache.org/jira/browse/IMPALA-7482
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: hang
> Attachments: vb1220-jstack2.out
>
>
> We've seen several instances of these mystery deadlocks in impalad's embedded
> JVM. The signature is a deadlock stemming from sun.security.provider.Sun
> being locked by an unknown owner.
> {noformat}
> Found one Java-level deadlock:
> =
> "Thread-24":
> waiting to lock monitor 0x12364688 (object 0x8027ef30, a
> sun.security.provider.Sun),
> which is held by UNKNOWN_owner_addr=0x14120800
> {noformat}
> If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck.
> If it happens in the Kudu client it also causes hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8252) Impala writes malformed thrift profiles
[
https://issues.apache.org/jira/browse/IMPALA-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-8252:
Description:
The change for IMPALA-1048 writes {{TRuntimeProfileNode.node_metadata}}
unconditionally, even when both its fields are unset. This trips up the Java
reader code, which expects to find exactly one type of a union to be set. The
resulting error looks like this:
{noformat}
Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized type 0
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:144)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at
com.cloudera.impala.thrift.TRuntimeProfileNodeMetadata.standardSchemeReadValue(TRuntimeProfileNodeMetadata.java:163)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:224)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:213)
at org.apache.thrift.TUnion.read(TUnion.java:138)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1532)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1341)
at
com.cloudera.impala.thrift.TRuntimeProfileNode.read(TRuntimeProfileNode.java:1187)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:426)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:405)
at
com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:339)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at com.cloudera.ipe.util.ThriftUtil.read(ThriftUtil.java:65)
{noformat}
(This particular implementation rewrites the Java namespace for compatibility
reasons, but the code used the latest master commit's thrift files.)
was:
The change for IMPALA-1048 writes TRuntimeProfileNode.meta_data
unconditionally, even when both its fields are unset. This trips up the Java
reader code, which expects to find exactly one type of a union to be set. The
resulting error looks like this:
{noformat}
Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized type 0
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:144)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at
com.cloudera.impala.thrift.TRuntimeProfileNodeMetadata.standardSchemeReadValue(TRuntimeProfileNodeMetadata.java:163)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:224)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:213)
at org.apache.thrift.TUnion.read(TUnion.java:138)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1532)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1341)
at
com.cloudera.impala.thrift.TRuntimeProfileNode.read(TRuntimeProfileNode.java:1187)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:426)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:405)
at
com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:339)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at com.cloudera.ipe.util.ThriftUtil.read(ThriftUtil.java:65)
{noformat}
(This particular implementation rewrites the Java namespace for compatibility
reasons, but the code used the latest master commit's thrift files.)
> Impala writes malformed thrift profiles
> ---
>
> Key: IMPALA-8252
> URL: https://issues.apache.org/jira/browse/IMPALA-8252
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Lars Volker
>Assignee: Lars Volker
>Priority: Blocker
> Labels: profile
>
> The change for IMPALA-1048 writes {{TRuntimeProfileNode.node_metadata}}
> unconditionally, even when both its fields are unset. This trips up the Java
> reader code, which expects to find exactly one type of a union to be set. The
> resulting error looks like this:
> {noformat}
> Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized type 0
> at
>
[jira] [Created] (IMPALA-8252) Impala writes malformed thrift profiles
Lars Volker created IMPALA-8252:
---
Summary: Impala writes malformed thrift profiles
Key: IMPALA-8252
URL: https://issues.apache.org/jira/browse/IMPALA-8252
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Lars Volker
Assignee: Lars Volker
The change for IMPALA-1048 writes TRuntimeProfileNode.meta_data
unconditionally, even when both its fields are unset. This trips up the Java
reader code, which expects to find exactly one type of a union to be set. The
resulting error looks like this:
{noformat}
Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized type 0
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:144)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at
com.cloudera.impala.thrift.TRuntimeProfileNodeMetadata.standardSchemeReadValue(TRuntimeProfileNodeMetadata.java:163)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:224)
at org.apache.thrift.TUnion$TUnionStandardScheme.read(TUnion.java:213)
at org.apache.thrift.TUnion.read(TUnion.java:138)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1532)
at
com.cloudera.impala.thrift.TRuntimeProfileNode$TRuntimeProfileNodeStandardScheme.read(TRuntimeProfileNode.java:1341)
at
com.cloudera.impala.thrift.TRuntimeProfileNode.read(TRuntimeProfileNode.java:1187)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:426)
at
com.cloudera.impala.thrift.TRuntimeProfileTree$TRuntimeProfileTreeStandardScheme.read(TRuntimeProfileTree.java:405)
at
com.cloudera.impala.thrift.TRuntimeProfileTree.read(TRuntimeProfileTree.java:339)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
at org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
at com.cloudera.ipe.util.ThriftUtil.read(ThriftUtil.java:65)
{noformat}
(This particular implementation rewrites the Java namespace for compatibility
reasons, but the code used the latest master commit's thrift files.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8069) crash in impala::Sorter::Run::Run
[
https://issues.apache.org/jira/browse/IMPALA-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778710#comment-16778710
]
ASF subversion and git services commented on IMPALA-8069:
-
Commit e4f46dad92f92c9a0e7730d7b393d42cf7ee8c7f in impala's branch
refs/heads/master from Paul Rogers
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e4f46da ]
IMPALA-8069: Crash in impala::Sorter::Run::Run
Modifies the planner to omit a sorter node in the obscure case we have
an analytic query with a constant sort. The analytic node requires a
buffered tuple. When the sort is omitted, creates a buffered tuple on
top of the node that would have been input to the sort.
Testing:
* Added a PlannerTest case.
* Added an end-to-end test.
* Reran the query that previously crashed: it now produces proper results.
Change-Id: I19a57f3791ce9e41fe0ba79dc5834e762341c74e
Reviewed-on: http://gerrit.cloudera.org:8080/12562
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> crash in impala::Sorter::Run::Run
> -
>
> Key: IMPALA-8069
> URL: https://issues.apache.org/jira/browse/IMPALA-8069
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Brown
>Assignee: Paul Rogers
>Priority: Blocker
> Labels: crash, query_generator, regression
>
> {noformat}
> 0 0x7fd37c9b8428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x7fd37c9ba02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x7fd37fb7fe49 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #3 0x7fd37fd35387 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #4 0x7fd37fb8995f in JVM_handle_linux_signal () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #5 0x7fd37fb7cf78 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x02723007 in impala::Sorter::Run::Run (this=0x13485200,
> parent=0xd3d8b00, sort_tuple_desc=0xf6dd320, initial_run=true) at
> /home/mikeb/Impala/be/src/runtime/sorter.cc:609
> #8 0x0272a3cb in impala::Sorter::Open (this=0xd3d8b00) at
> /home/mikeb/Impala/be/src/runtime/sorter.cc:1514
> #9 0x0237b186 in impala::SortNode::Open (this=0xd3d8840,
> state=0xffa1ba0) at /home/mikeb/Impala/be/src/exec/sort-node.cc:84
> #10 0x0239b44b in impala::AnalyticEvalNode::Open (this=0x100aba80,
> state=0xffa1ba0) at /home/mikeb/Impala/be/src/exec/analytic-eval-node.cc:182
> #11 0x01f20cba in impala::FragmentInstanceState::Open
> (this=0xf5afd80) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:304
> #12 0x01f1deee in impala::FragmentInstanceState::Exec
> (this=0xf5afd80) at
> /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:84
> #13 0x01f2ea39 in impala::QueryState::ExecFInstance (this=0xea2,
> fis=0xf5afd80) at /home/mikeb/Impala/be/src/runtime/query-state.cc:584
> #14 0x01f2cd42 in impala::QueryStateoperator()(void)
> const (__closure=0x7fd2b1869ca8) at
> /home/mikeb/Impala/be/src/runtime/query-state.cc:493
> #15 0x01f2f7e1 in
> boost::detail::function::void_function_obj_invoker0,
> void>::invoke(boost::detail::function::function_buffer &) (
> function_obj_ptr=...) at
> /home/mikeb/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #16 0x01d46c96 in boost::function0::operator()
> (this=0x7fd2b1869ca0) at
> /home/mikeb/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
> #17 0x021d4897 in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) (name=..., category=...,
> functor=..., parent_thread_info=0x7fd2b206a950,
> thread_started=0x7fd2b20698f0) at /home/mikeb/Impala/be/src/util/thread.cc:359
> #18 0x021dcbb7 in boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> (this=0xed511c0,
> f=@0xed511b8: 0x21d4530 const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*)>, a=...) at
> /home/mikeb/Impala/toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525
> #19 0x021dcadb in boost::_bi::bind_t const&, std::string const&, boost::function, impala::ThreadDebugInfo
> const*, impala::Promise*),
>
[jira] [Commented] (IMPALA-8245) Add hostname to timeout error message in HdfsMonitoredOps
[ https://issues.apache.org/jira/browse/IMPALA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778709#comment-16778709 ] ASF subversion and git services commented on IMPALA-8245: - Commit 72fd7c3d4d50f30479b5f6d50b327dff9ff013ac in impala's branch refs/heads/master from poojanilangekar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=72fd7c3 ] IMPALA-8245: Add hostname to the error message in HdfsMonitoredOps When a task submitted via HdfsMonitoredOps fails, it raises TErrorCode::THREAD_POOL_TASK_TIMED_OUT or TErrorCode::THREAD_POOL_SUBMIT_FAILED errors. The error generated calls GetDescription() to provide information about the failed hdfs operation. This change appends the hostname to the description to enable the user to easily identify the host which has reached a bad connection state with the NameNode. Testing: Modified the test_hdfs_timeout.py to ensure that the hostname is logged in the error message. Change-Id: Ief1e21560b6fb54965f2fb2793c32c2ba5176ba2 Reviewed-on: http://gerrit.cloudera.org:8080/12593 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add hostname to timeout error message in HdfsMonitoredOps > - > > Key: IMPALA-8245 > URL: https://issues.apache.org/jira/browse/IMPALA-8245 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Pooja Nilangekar >Priority: Major > > If a DiskIo operation times out, it generates a > TErrorCode::THREAD_POOL_TASK_TIMED_OUT or > TErrorCode::THREAD_POOL_SUBMIT_FAILED error codes. These call > GetDescription() to get DiskIo related context. That information should > include the hostname where the error occurred to allow tracking down a > problematic host that is seeing DiskIo timeouts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8181) Show abbreviated row counts in DESCRIBE output
[ https://issues.apache.org/jira/browse/IMPALA-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778711#comment-16778711 ] ASF subversion and git services commented on IMPALA-8181: - Commit 360f88e20709a4f1447c4997d1dc458e80874389 in impala's branch refs/heads/master from Paul Rogers [ https://gitbox.apache.org/repos/asf?p=impala.git;h=360f88e ] IMPALA-8181: Abbreviate row counts in EXPLAIN A recent fix added node cardinality to the standard EXPLAIN output, displaying a large number like 123456780 as 123.46M. This patch applies the same fix to the remaining row count numbers: metadata, extrapolated rows, etc. Tests: * Rebased PlannerTest .test files as needed for the new row count format. * Reran all tests to check for dependencies on the old format. Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c Reviewed-on: http://gerrit.cloudera.org:8080/12438 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Show abbreviated row counts in DESCRIBE output > -- > > Key: IMPALA-8181 > URL: https://issues.apache.org/jira/browse/IMPALA-8181 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > > A recent change added plan node cardinality to the DESCRIBE output using a > "metric" format: 123.46M instead of 123456789. This makes the output easier > to read. Also, since all numbers are estimate, having seven or eight digits > of precision is not very helpful. > This ticket requests that the same formatting be used for the other places > where the DESCRIBE output shows row counts: table stats, extrapolated row > count, partition row counts, and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8188) Some SSDs are not properly detected as non-rotational
[
https://issues.apache.org/jira/browse/IMPALA-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778712#comment-16778712
]
ASF subversion and git services commented on IMPALA-8188:
-
Commit 7ba6aa64a2455df30524ad42b10d7fd9227efacb in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7ba6aa6 ]
IMPALA-8188: Fix DiskInfo::GetDeviceNames() for NVME disks
DiskInfo::GetDeviceNames() assumes a device name pattern where
the last digits can be removed to get the main disk device
(i.e. sda3 -> sda). NVME devices have a naming pattern of
nvme{device_id}n{namespace_id}p{partition_id} where the main
disk device removes the p{partition_id} part. This difference
in naming prevents DiskInfo::GetDeviceNames() from looking
up the associated information in /sys/block to find whether
it is a rotational device (and thus use an appropriate
number of DiskIo threads).
This adds a condition to detect an NVME device name and
handle it correctly.
Testing:
- Added unit test for the function used for detection
- Tested this on an AWS m5-4xlarge, which is a Nitro instance
type that exposes the main disk as an NVME device.
Change-Id: I4d5bd93b4b09682a8c8248e7aa123d23d27cfeb4
Reviewed-on: http://gerrit.cloudera.org:8080/12557
Tested-by: Impala Public Jenkins
Reviewed-by: Philip Zeyliger
> Some SSDs are not properly detected as non-rotational
> -
>
> Key: IMPALA-8188
> URL: https://issues.apache.org/jira/browse/IMPALA-8188
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Critical
>
> Here is an example Impala log:
>
> {noformat}
> I0211 10:50:40.650727 18344 init.cc:288] Disk Info:
> Num disks 2:
> nvme0n (rotational=true)
> nvme0n1p (rotational=true){noformat}
> I logged into an equivalent machine, and the OS sees these as not rotational:
>
>
> {noformat}
> # cat /sys/block/nvme0n1/queue/rotational
> 0
> {noformat}
> Device names that end in a number get trimmed (i.e. /dev/sda2 becomes
> /dev/sda). See
> [https://github.com/apache/impala/blob/master/be/src/util/disk-info.cc#L73-L74]
> These devices don't follow that pattern, so we don't find the right files.
> Neither /sys/block/nvme0n nor /sys/block/nvme0n1p exist, so both fall back to
> being rotational.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
[
https://issues.apache.org/jira/browse/IMPALA-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778695#comment-16778695
]
Michael Ho commented on IMPALA-8251:
{{--stress_datastream_recvr_delay_ms}} appears to be debug only. Will mark the
test to skip on non-debug build. Mostly a test bug.
> Impala fails to start for
> TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
> --
>
> Key: IMPALA-8251
> URL: https://issues.apache.org/jira/browse/IMPALA-8251
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
>
> custom_cluster/test_exchange_deferred_batches.py is hitting the following
> error on release builds:
> {noformat}
> ERROR at setup of
> TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> common/custom_cluster_test_suite.py:155: in setup_method
> expected_num_executors=cluster_size)
> common/custom_cluster_test_suite.py:230: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> /usr/lib/python2.7/subprocess.py:541: in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
> '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
> '--impalad_args=--stress_datastream_recvr_delay_ms=3000
> --exchg_node_buffer_size_bytes=1024
> --datastream_service_num_deserialization_threads=1 ',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50 ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> The impalad logs show errors:
> {noformat}
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address
> for: jhzvlthd
> . Impalad exiting.
> {noformat}
> It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
[
https://issues.apache.org/jira/browse/IMPALA-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-8251:
--
Assignee: Michael Ho
> Impala fails to start for
> TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
> --
>
> Key: IMPALA-8251
> URL: https://issues.apache.org/jira/browse/IMPALA-8251
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
>
> custom_cluster/test_exchange_deferred_batches.py is hitting the following
> error on release builds:
> {noformat}
> ERROR at setup of
> TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> common/custom_cluster_test_suite.py:155: in setup_method
> expected_num_executors=cluster_size)
> common/custom_cluster_test_suite.py:230: in _start_impala_cluster
> check_call(cmd + options, close_fds=True)
> /usr/lib/python2.7/subprocess.py:541: in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command
> '['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
> '--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
> '--impalad_args=--stress_datastream_recvr_delay_ms=3000
> --exchg_node_buffer_size_bytes=1024
> --datastream_service_num_deserialization_threads=1 ',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50 ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> The impalad logs show errors:
> {noformat}
> Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
> F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address
> for: jhzvlthd
> . Impalad exiting.
> {noformat}
> It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
Joe McDonnell created IMPALA-8251:
-
Summary: Impala fails to start for
TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
Key: IMPALA-8251
URL: https://issues.apache.org/jira/browse/IMPALA-8251
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell
custom_cluster/test_exchange_deferred_batches.py is hitting the following error
on release builds:
{noformat}
ERROR at setup of
TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none]
common/custom_cluster_test_suite.py:155: in setup_method
expected_num_executors=cluster_size)
common/custom_cluster_test_suite.py:230: in _start_impala_cluster
check_call(cmd + options, close_fds=True)
/usr/lib/python2.7/subprocess.py:541: in check_call
raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command
'['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
'--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
'--impalad_args=--stress_datastream_recvr_delay_ms=3000
--exchg_node_buffer_size_bytes=1024
--datastream_service_num_deserialization_threads=1 ',
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50 ',
'--impalad_args=--default_query_options=']' returned non-zero exit status 1
{noformat}
The impalad logs show errors:
{noformat}
Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address for:
jhzvlthd
. Impalad exiting.
{noformat}
It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8251) Impala fails to start for TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
Joe McDonnell created IMPALA-8251:
-
Summary: Impala fails to start for
TestExchangeDeferredBatches.test_exchange_small_buffer() under release build
Key: IMPALA-8251
URL: https://issues.apache.org/jira/browse/IMPALA-8251
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Joe McDonnell
custom_cluster/test_exchange_deferred_batches.py is hitting the following error
on release builds:
{noformat}
ERROR at setup of
TestExchangeDeferredBatches.test_exchange_small_buffer[protocol: beeswax |
exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
text/none]
common/custom_cluster_test_suite.py:155: in setup_method
expected_num_executors=cluster_size)
common/custom_cluster_test_suite.py:230: in _start_impala_cluster
check_call(cmd + options, close_fds=True)
/usr/lib/python2.7/subprocess.py:541: in check_call
raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command
'['/home/joe/view2/Impala/bin/start-impala-cluster.py', '--cluster_size=3',
'--num_coordinators=3', '--log_dir=/tmp/', '--log_level=1',
'--impalad_args=--stress_datastream_recvr_delay_ms=3000
--exchg_node_buffer_size_bytes=1024
--datastream_service_num_deserialization_threads=1 ',
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50 ',
'--impalad_args=--default_query_options=']' returned non-zero exit status 1
{noformat}
The impalad logs show errors:
{noformat}
Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
F0226 08:11:15.563117 357 impalad-main.cc:72] Could not find IPv4 address for:
jhzvlthd
. Impalad exiting.
{noformat}
It reproduces consistently on my local desktop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8250) Impala crashes with -Xcheck:jni
[
https://issues.apache.org/jira/browse/IMPALA-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778687#comment-16778687
]
Philip Zeyliger commented on IMPALA-8250:
-
The first few of these were easy and https://gerrit.cloudera.org/#/c/12582/
captures them.
To look at the rest, I've been compiling OpenJDK8
(http://cr.openjdk.java.net/~ihse/demo-new-build-readme/common/doc/building.html#tldr-instructions-for-the-impatient
is surprisingly reasonably) with the following patch applied to the hotspot
subdirectory:
{code}
$hg diff
diff -r 76a9c9cf14f1 src/share/vm/prims/jniCheck.cpp
--- a/src/share/vm/prims/jniCheck.cpp Tue Jan 15 10:43:31 2019 +
+++ b/src/share/vm/prims/jniCheck.cpp Tue Feb 26 15:22:56 2019 -0800
@@ -143,11 +143,18 @@
static const char * fatal_instance_field_mismatch = "Field type (instance)
mismatch in JNI get/set field operations";
static const char * fatal_non_string = "JNI string operation received a
non-string";
+// PHIL
+static inline void dump_native_stack(JavaThread* thr) {
+ frame fr = os::current_frame();
+ char buf[6000];
+ print_native_stack(tty, fr, thr, buf, sizeof(buf));
+}
// When in VM state:
static void ReportJNIWarning(JavaThread* thr, const char *msg) {
tty->print_cr("WARNING in native method: %s", msg);
thr->print_stack();
+ dump_native_stack(thr);
}
// When in NATIVE state:
@@ -199,11 +206,15 @@
tty->print_cr("WARNING in native method: JNI call made without checking
exceptions when required to from %s",
thr->get_pending_jni_exception_check());
thr->print_stack();
+
+ dump_native_stack(thr);
)
thr->clear_pending_jni_exception_check(); // Just complain once
}
}
+
+
/**
* Add to the planned number of handles. I.e. plus current live & warning
threshold
*/
@@ -254,6 +265,7 @@
tty->print_cr("WARNING: JNI local refs: %zu, exceeds capacity: %zu",
live_handles, planned_capacity);
thr->print_stack();
+ dump_native_stack(thr);
)
// Complain just the once, reset to current + warn threshold
add_planned_handle_capacity(handles, 0);
{code}
This told me that the vast majority of issues are in Impala's HBase code.
> Impala crashes with -Xcheck:jni
> ---
>
> Key: IMPALA-8250
> URL: https://issues.apache.org/jira/browse/IMPALA-8250
> Project: IMPALA
> Issue Type: Task
>Reporter: Philip Zeyliger
>Priority: Major
>
> The JVM has a checker for JNI usage, and Impala (and libhdfs) have some
> violations. This ticket captures figuring that out. At least one of the
> issues can crash Impala.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8163) Local catalog mode needs to be visible on catalogd web UI if turned on
[ https://issues.apache.org/jira/browse/IMPALA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-8163. -- Resolution: Fixed Fix Version/s: Impala 3.2.0 Change committed to master. > Local catalog mode needs to be visible on catalogd web UI if turned on > --- > > Key: IMPALA-8163 > URL: https://issues.apache.org/jira/browse/IMPALA-8163 > Project: IMPALA > Issue Type: Bug >Reporter: Adrian Ng >Assignee: Anurag Mantripragada >Priority: Major > Fix For: Impala 3.2.0 > > > Once local catalog mode (i.e. metadata v2) is turned on (off by default), it > needs to be visible on catalogd web UI because it is an experimental protocol > so admins need to be aware of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8163) Local catalog mode needs to be visible on catalogd web UI if turned on
[ https://issues.apache.org/jira/browse/IMPALA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-8163. -- Resolution: Fixed Fix Version/s: Impala 3.2.0 Change committed to master. > Local catalog mode needs to be visible on catalogd web UI if turned on > --- > > Key: IMPALA-8163 > URL: https://issues.apache.org/jira/browse/IMPALA-8163 > Project: IMPALA > Issue Type: Bug >Reporter: Adrian Ng >Assignee: Anurag Mantripragada >Priority: Major > Fix For: Impala 3.2.0 > > > Once local catalog mode (i.e. metadata v2) is turned on (off by default), it > needs to be visible on catalogd web UI because it is an experimental protocol > so admins need to be aware of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7935) /catalog_object end point broken in LocalCatalog mode
[
https://issues.apache.org/jira/browse/IMPALA-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7935 started by Anurag Mantripragada.
> /catalog_object end point broken in LocalCatalog mode
> -
>
> Key: IMPALA-7935
> URL: https://issues.apache.org/jira/browse/IMPALA-7935
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: bharath v
>Assignee: Anurag Mantripragada
>Priority: Major
> Labels: observability
>
> Start Impala coordinator in LocalCatalog mode (v2)
> {code}
> start-impala-cluster.py -s 1 --impalad_args="--use_local_catalog=true"
> --catalogd_args="--catalog_topic_mode=minimal"
> {code}
> Check the following URL.
> http://:25000/catalog_object?object_type=TABLE_name=functional_text_lzo.alltypesaggmultifiles
> It says,
> {noformat}
> Error: UnsupportedOperationException: LocalCatalog.getTCatalogObject
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8239) Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
[
https://issues.apache.org/jira/browse/IMPALA-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8239.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ +
> num_pending_enqueue_) > 0
>
>
> Key: IMPALA-8239
> URL: https://issues.apache.org/jira/browse/IMPALA-8239
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
> Labels: crash
> Fix For: Impala 3.2.0
>
> Attachments: IMPALA-8239-logs.tar.gz
>
>
> {noformat}
> F0221 12:36:27.886497 157443 krpc-data-stream-recvr.cc:234]
> 3843237c92f4c6b5:b5aa170b0092] Check failed: deferred_rpcs_.empty() ||
> (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
> *** Check failure stack trace: ***
> @ 0x47eb92c google::LogMessage::Fail()
> @ 0x47ed1d1 google::LogMessage::SendToLog()
> @ 0x47eb306 google::LogMessage::Flush()
> @ 0x47ee8cd google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ed07a0 impala::KrpcDataStreamRecvr::SenderQueue::GetBatch()
> @ 0x1ed65e7 impala::KrpcDataStreamRecvr::GetBatch()
> @ 0x22c5d67 impala::ExchangeNode::FillInputRowBatch()
> @ 0x22c5953 impala::ExchangeNode::Open()
> @ 0x240079a
> impala::BlockingJoinNode::ProcessBuildInputAndOpenProbe()
> @ 0x238720f impala::PartitionedHashJoinNode::Open()
> @ 0x23e6d76 impala::AggregationNode::Open()
> @ 0x1f52b2f impala::FragmentInstanceState::Open()
> @ 0x1f4f7f3 impala::FragmentInstanceState::Exec()
> @ 0x1f62dd9 impala::QueryState::ExecFInstance()
> @ 0x1f610bb
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1f6421a
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1d766b3 boost::function0<>::operator()()
> @ 0x2223e4e impala::Thread::SuperviseThread()
> @ 0x222c1d2 boost::_bi::list5<>::operator()<>()
> @ 0x222c0f6 boost::_bi::bind_t<>::operator()()
> @ 0x222c0b9 boost::detail::thread_data<>::run()
> @ 0x3715469 thread_proxy
> @ 0x7f262e1efe24 start_thread
> @ 0x7f262df1d34c __clone
> {noformat}
> Immediately before it in the log a different thread printed:
> {noformat}
> E0221 12:36:27.557099 157280 krpc-data-stream-sender.cc:344]
> 4145938c29f31877:6ed16e5e022d] channel send to 10.17.211.20:27000 failed:
> (fragment_instance_id=4145938c29f31877:6ed16e5e024c): Memory limit
> exceeded: Failed to allocate row batch
> EXCHANGE_NODE (id=189) could not allocate 8.00 KB without exceeding limit.
> Error occurred on backend vc0510.halxg.cloudera.com:22000
> Memory left in process limit: 12.11 GB
> Memory left in query limit: 541.65 MB
> Query(4145938c29f31877:6ed16e5e): Limit=1.39 GB Reservation=704.88 MB
> ReservationLimit=1.11 GB OtherMemory=174.47 MB Total=879.35 MB Peak=926.55 MB
> Unclaimed reservations: Reservation=430.88 MB OtherMemory=0 Total=430.88 MB
> Peak=740.31 MB
> : Reservation=0 OtherMemory=0 Total=0 Peak=1.59 MB
> SORT_NODE (id=118): Total=0 Peak=4.00 KB
> AGGREGATION_NODE (id=203): Reservation=0 OtherMemory=0 Total=0 Peak=81.12
> KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> EXCHANGE_NODE (id=202): Reservation=0 OtherMemory=0 Total=0 Peak=0
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=204): Total=0 Peak=1.75 KB
> CodeGen: Total=0 Peak=1.50 MB
> : Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> AGGREGATION_NODE (id=117): Reservation=0 OtherMemory=0 Total=0 Peak=81.12
> KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> UNION_NODE (id=0): Total=0 Peak=0
> AGGREGATION_NODE (id=144): Reservation=0 OtherMemory=0 Total=0 Peak=34.12
> KB
> GroupingAggregator 0: Total=0 Peak=34.12 KB
> {noformat}
> Logs for the crashing thread:
> {noformat}
> I0221 12:36:23.677255 157443 query-state.cc:624]
> 3843237c92f4c6b5:b5aa170b0092] Executing instance.
> instance_id=3843237c92f4c6b5:b5aa170b0092 fragment_idx=10
> per_fragment_instance_idx=7 coord_state_idx=10
> #in-flight=525
> {noformat}
> The query was:
> {noformat}
> I0221 12:36:18.419795 140378 impala-server.cc:1063]
> 3843237c92f4c6b5:b5aa170b] Registered query
> query_id=3843237c92f4c6b5:b5aa170b
> session_id=f24bd763b83e0b5e:77f6ea4b07165d9b
> I0221
[jira] [Resolved] (IMPALA-8239) Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
[
https://issues.apache.org/jira/browse/IMPALA-8239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8239.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> Check failed: deferred_rpcs_.empty() || (num_deserialize_tasks_pending_ +
> num_pending_enqueue_) > 0
>
>
> Key: IMPALA-8239
> URL: https://issues.apache.org/jira/browse/IMPALA-8239
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
> Labels: crash
> Fix For: Impala 3.2.0
>
> Attachments: IMPALA-8239-logs.tar.gz
>
>
> {noformat}
> F0221 12:36:27.886497 157443 krpc-data-stream-recvr.cc:234]
> 3843237c92f4c6b5:b5aa170b0092] Check failed: deferred_rpcs_.empty() ||
> (num_deserialize_tasks_pending_ + num_pending_enqueue_) > 0
> *** Check failure stack trace: ***
> @ 0x47eb92c google::LogMessage::Fail()
> @ 0x47ed1d1 google::LogMessage::SendToLog()
> @ 0x47eb306 google::LogMessage::Flush()
> @ 0x47ee8cd google::LogMessageFatal::~LogMessageFatal()
> @ 0x1ed07a0 impala::KrpcDataStreamRecvr::SenderQueue::GetBatch()
> @ 0x1ed65e7 impala::KrpcDataStreamRecvr::GetBatch()
> @ 0x22c5d67 impala::ExchangeNode::FillInputRowBatch()
> @ 0x22c5953 impala::ExchangeNode::Open()
> @ 0x240079a
> impala::BlockingJoinNode::ProcessBuildInputAndOpenProbe()
> @ 0x238720f impala::PartitionedHashJoinNode::Open()
> @ 0x23e6d76 impala::AggregationNode::Open()
> @ 0x1f52b2f impala::FragmentInstanceState::Open()
> @ 0x1f4f7f3 impala::FragmentInstanceState::Exec()
> @ 0x1f62dd9 impala::QueryState::ExecFInstance()
> @ 0x1f610bb
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1f6421a
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1d766b3 boost::function0<>::operator()()
> @ 0x2223e4e impala::Thread::SuperviseThread()
> @ 0x222c1d2 boost::_bi::list5<>::operator()<>()
> @ 0x222c0f6 boost::_bi::bind_t<>::operator()()
> @ 0x222c0b9 boost::detail::thread_data<>::run()
> @ 0x3715469 thread_proxy
> @ 0x7f262e1efe24 start_thread
> @ 0x7f262df1d34c __clone
> {noformat}
> Immediately before it in the log a different thread printed:
> {noformat}
> E0221 12:36:27.557099 157280 krpc-data-stream-sender.cc:344]
> 4145938c29f31877:6ed16e5e022d] channel send to 10.17.211.20:27000 failed:
> (fragment_instance_id=4145938c29f31877:6ed16e5e024c): Memory limit
> exceeded: Failed to allocate row batch
> EXCHANGE_NODE (id=189) could not allocate 8.00 KB without exceeding limit.
> Error occurred on backend vc0510.halxg.cloudera.com:22000
> Memory left in process limit: 12.11 GB
> Memory left in query limit: 541.65 MB
> Query(4145938c29f31877:6ed16e5e): Limit=1.39 GB Reservation=704.88 MB
> ReservationLimit=1.11 GB OtherMemory=174.47 MB Total=879.35 MB Peak=926.55 MB
> Unclaimed reservations: Reservation=430.88 MB OtherMemory=0 Total=430.88 MB
> Peak=740.31 MB
> : Reservation=0 OtherMemory=0 Total=0 Peak=1.59 MB
> SORT_NODE (id=118): Total=0 Peak=4.00 KB
> AGGREGATION_NODE (id=203): Reservation=0 OtherMemory=0 Total=0 Peak=81.12
> KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> EXCHANGE_NODE (id=202): Reservation=0 OtherMemory=0 Total=0 Peak=0
> KrpcDeferredRpcs: Total=0 Peak=0
> KrpcDataStreamSender (dst_id=204): Total=0 Peak=1.75 KB
> CodeGen: Total=0 Peak=1.50 MB
> : Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> AGGREGATION_NODE (id=117): Reservation=0 OtherMemory=0 Total=0 Peak=81.12
> KB
> GroupingAggregator 0: Total=0 Peak=81.12 KB
> UNION_NODE (id=0): Total=0 Peak=0
> AGGREGATION_NODE (id=144): Reservation=0 OtherMemory=0 Total=0 Peak=34.12
> KB
> GroupingAggregator 0: Total=0 Peak=34.12 KB
> {noformat}
> Logs for the crashing thread:
> {noformat}
> I0221 12:36:23.677255 157443 query-state.cc:624]
> 3843237c92f4c6b5:b5aa170b0092] Executing instance.
> instance_id=3843237c92f4c6b5:b5aa170b0092 fragment_idx=10
> per_fragment_instance_idx=7 coord_state_idx=10
> #in-flight=525
> {noformat}
> The query was:
> {noformat}
> I0221 12:36:18.419795 140378 impala-server.cc:1063]
> 3843237c92f4c6b5:b5aa170b] Registered query
> query_id=3843237c92f4c6b5:b5aa170b
> session_id=f24bd763b83e0b5e:77f6ea4b07165d9b
> I0221
[jira] [Created] (IMPALA-8250) Impala crashes with -Xcheck:jni
Philip Zeyliger created IMPALA-8250: --- Summary: Impala crashes with -Xcheck:jni Key: IMPALA-8250 URL: https://issues.apache.org/jira/browse/IMPALA-8250 Project: IMPALA Issue Type: Task Reporter: Philip Zeyliger The JVM has a checker for JNI usage, and Impala (and libhdfs) have some violations. This ticket captures figuring that out. At least one of the issues can crash Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8250) Impala crashes with -Xcheck:jni
Philip Zeyliger created IMPALA-8250: --- Summary: Impala crashes with -Xcheck:jni Key: IMPALA-8250 URL: https://issues.apache.org/jira/browse/IMPALA-8250 Project: IMPALA Issue Type: Task Reporter: Philip Zeyliger The JVM has a checker for JNI usage, and Impala (and libhdfs) have some violations. This ticket captures figuring that out. At least one of the issues can crash Impala. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8247) Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are not running
[ https://issues.apache.org/jira/browse/IMPALA-8247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778242#comment-16778242 ] ASF subversion and git services commented on IMPALA-8247: - Commit e9936b02cf5b9673d6dd8f3624b07e2ec81f763e in impala's branch refs/heads/master from Joe McDonnell [ https://gitbox.apache.org/repos/asf?p=impala.git;h=e9936b0 ] IMPALA-8247: Fix tests missing from unified backend executable When converting the tests in be/src/util to the unified backend test executable in IMPALA-8071, some tests did not get linked into the executable appropriately. This means that they stopped running. Specifically, the tests in bit-stream-utils-test.cc and system-state-info-test.cc were left out. This adds them back and modifies bin/validate-unified-backend-filters.py to check both directions: - If a filter is specified, it must match a test in the executable. - If the executable has a test, some filter must match it. I ran the backend tests on debug and ASAN with this change. Change-Id: I80d8b5779cb0ac663e32c72722e0c73b68a43d2e Reviewed-on: http://gerrit.cloudera.org:8080/12584 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong > Backend tests from bit-stream-utils-test.cc and system-state-info-test.cc are > not running > - > > Key: IMPALA-8247 > URL: https://issues.apache.org/jira/browse/IMPALA-8247 > Project: IMPALA > Issue Type: Bug > Components: Backend, Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Blocker > Labels: broken-build > > Several tests in be/src/util were converted to the unified backend test > executable in IMPALA-8071. Unfortunately, some tests stopped running as part > of this change. Specifically, the tests in bit-stream-utils-test.cc and > system-state-info-test.cc were not included in the test binary and stopped > running. The tests changed to use ADD_UNIFIED_BE_LSAN_TEST, but the files > were not included in the UtilTests library. > These two files should be included in the test executable and we need to > update the validation of the unified test executable > (bin/validate-unified-backend-test-filters.py) to catch this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7979) Enhance decoders to support value-skipping
[ https://issues.apache.org/jira/browse/IMPALA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778241#comment-16778241 ] ASF subversion and git services commented on IMPALA-7979: - Commit 5a270e097214624ced7814692c58b03d575e7ace in impala's branch refs/heads/master from Zoltan Borok-Nagy [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5a270e0 ] IMPALA-7979: Fix bug of SkipValues() and GetNextValues() interaction Alternating calls to DictDecoder's SkipValues() and GetNextValues() could result in a decoding error. RleBatchDecoder::DecodeLiteralValues() returned false when invoked with num_literals_to_consume=0 and had buffered literals. Then the caller (DictDecoder::GetNextValues()) thought it failed, and in the end it resulted in a decoding error. It could only occur if GetNextValues() was invoked after SkipValues(). Otherwise the RleBatchDecoder in DictDecoder could not have buffered literals, since they were buffered in the dictionary decoder. Since SkipValues() is not used yet, it couldn't happen in production code. I added a backend test to fuzz test the interaction between SkipValues() and GetNextValues(). I looped the test for a few hours without seeing an error. Change-Id: Id0426cdef54ee9bc2306a542098a92c640dc41c4 Reviewed-on: http://gerrit.cloudera.org:8080/12555 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Enhance decoders to support value-skipping > -- > > Key: IMPALA-7979 > URL: https://issues.apache.org/jira/browse/IMPALA-7979 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Fix For: Impala 3.2.0 > > > When the parquet scanner can use the page index it is sometimes needed to > skip values from the pages. We can just drop these values since we'll never > need them. > The decoders don't support value-skipping at the moment, therefore we have to > decode each and every value even if we know that we will never need many of > them. > The decoders should have methods for value-skipping to make scanning more > efficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8241) from_utc_timestamp returns inconsistent results with Hive
[
https://issues.apache.org/jira/browse/IMPALA-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778150#comment-16778150
]
Attila Jeges commented on IMPALA-8241:
--
[~stiga-huang] [~tarmstrong] There's a difference between how Impala and Hive
interpret "cast(nanosecs as timestamp)". By default Impala returns the result
in UTC, whereas Hive's result is in the local timezone.
E.g..:
select cast(0 as timestamp);
returns "1970-01-01 00:00:00" in Impala and "1970-01-01 01:00:00" in Hive on my
machine (local timezone is set to Europe/Budapest).
As [~tarmstrong] mentioned the '-use_local_tz_for_unix_timestamp_conversions'
flag can make Impala behave like Hive.
> from_utc_timestamp returns inconsistent results with Hive
> -
>
> Key: IMPALA-8241
> URL: https://issues.apache.org/jira/browse/IMPALA-8241
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
>
> This can be reproduced in both master and 2.x branches.
> {code}
> [localhost:21000] default> select from_utc_timestamp(cast(40 * 3600.0 as
> timestamp), 'EST');
> Query: select from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'EST')
> Query submitted at: 2019-02-23 17:27:02 (Coordinator:
> http://impala-jenkins-slave-02:25000)
> Query progress can be monitored at:
> http://impala-jenkins-slave-02:25000/query_plan?query_id=f476c87a904f281:71588a24
> +---+
> | from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'est') |
> +---+
> | 2015-08-19 11:00:00 |
> +---+
> Fetched 1 row(s) in 0.64s
> {code}
> {code}
> hive> select from_utc_timestamp(cast(40 * 3600.0 as timestamp), 'EST');
> OK
> 2015-08-19 04:00:00
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7972) Detect self-events to avoid unnecessary invalidates
[ https://issues.apache.org/jira/browse/IMPALA-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778102#comment-16778102 ] Vihang Karajgaonkar commented on IMPALA-7972: - Adding gerrit code review link > Detect self-events to avoid unnecessary invalidates > --- > > Key: IMPALA-7972 > URL: https://issues.apache.org/jira/browse/IMPALA-7972 > Project: IMPALA > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > When a metastore objects are created/altered or dropped, it would generate a > event which will be polled by the same Catalog server. Such self-events > should be detected and we should avoid invalidating the tables when they are > received. See the design doc attached to the main JIRA for more details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8249) End-to-end test framework doesn't read aggregated counters properly
[
https://issues.apache.org/jira/browse/IMPALA-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy reassigned IMPALA-8249:
-
Assignee: Zoltán Borók-Nagy
> End-to-end test framework doesn't read aggregated counters properly
> ---
>
> Key: IMPALA-8249
> URL: https://issues.apache.org/jira/browse/IMPALA-8249
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>
> The test framework doesn't always read the correct value of counters from the
> runtime profile. In the .test files we can have a RUNTIME_PROFILE section
> where we can test our expectations against runtime profile data. We can even
> calculate aggregates of runtime data, currently only SUM is supported over
> integer data, e.g.:
>
> {code:java}
> RUNTIME_PROFILE
> aggregation(SUM, RowsReturned): 2142543
> {code}
>
> However, the counters are pretty-printed in the runtime profile, which means
> that if they are greater than 1000, a shortened version is printed first,
> then the accurate number comes in parenthesis , e.g.:
>
> {code:java}
> RowsReturned: 2.14M (2142543){code}
>
> When the test framework parses the value of an aggregated counter, it wrongly
> tries to parse the short version as a number, which returns a wrong value (2
> instead of 2142543 in the example).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8249) End-to-end test framework doesn't read aggregated counters properly
[
https://issues.apache.org/jira/browse/IMPALA-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-8249:
--
Description:
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
was:
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:(
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
> End-to-end test framework doesn't read aggregated counters properly
> ---
>
> Key: IMPALA-8249
> URL: https://issues.apache.org/jira/browse/IMPALA-8249
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>
> The test framework doesn't always read the correct value of counters from the
> runtime profile. In the .test files we can have a RUNTIME_PROFILE section
> where we can test our expectations against runtime profile data. We can even
> calculate aggregates of runtime data, currently only SUM is supported over
> integer data, e.g.:
> {code:java}
> RUNTIME_PROFILE
> aggregation(SUM, RowsReturned): 2142543
> {code}
> However, the counters are pretty-printed in the runtime profile, which means
> that if they are greater than 1000, a shortened version is printed first,
> then the accurate number comes in parenthesis , e.g.:
> {code:java}
> RowsReturned: 2.14M (2142543){code}
> When the test framework parses the value of an aggregated counter, it
> wrongly tries to parse the short version as a number, which returns a wrong
> value (2 instead of 2142543 in the example).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8249) End-to-end test framework doesn't read aggregated counters properly
[
https://issues.apache.org/jira/browse/IMPALA-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-8249:
--
Description:
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:(
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
was:
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
> End-to-end test framework doesn't read aggregated counters properly
> ---
>
> Key: IMPALA-8249
> URL: https://issues.apache.org/jira/browse/IMPALA-8249
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
>
> The test framework doesn't always read the correct value of counters from the
> runtime profile. In the .test files we can have a RUNTIME_PROFILE section
> where we can test our expectations against runtime profile data. We can even
> calculate aggregates of runtime data, currently only SUM is supported over
> integer data, e.g.:(
> {code:java}
> RUNTIME_PROFILE
> aggregation(SUM, RowsReturned): 2142543
> {code}
> However, the counters are pretty-printed in the runtime profile, which means
> that if they are greater than 1000, a shortened version is printed first,
> then the accurate number comes in parenthesis , e.g.:
> {code:java}
> RowsReturned: 2.14M (2142543){code}
> When the test framework parses the value of an aggregated counter, it
> wrongly tries to parse the short version as a number, which returns a wrong
> value (2 instead of 2142543 in the example).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8249) End-to-end test framework doesn't read aggregated counters properly
Zoltán Borók-Nagy created IMPALA-8249:
-
Summary: End-to-end test framework doesn't read aggregated
counters properly
Key: IMPALA-8249
URL: https://issues.apache.org/jira/browse/IMPALA-8249
Project: IMPALA
Issue Type: Bug
Reporter: Zoltán Borók-Nagy
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8249) End-to-end test framework doesn't read aggregated counters properly
Zoltán Borók-Nagy created IMPALA-8249:
-
Summary: End-to-end test framework doesn't read aggregated
counters properly
Key: IMPALA-8249
URL: https://issues.apache.org/jira/browse/IMPALA-8249
Project: IMPALA
Issue Type: Bug
Reporter: Zoltán Borók-Nagy
The test framework doesn't always read the correct value of counters from the
runtime profile. In the .test files we can have a RUNTIME_PROFILE section where
we can test our expectations against runtime profile data. We can even
calculate aggregates of runtime data, currently only SUM is supported over
integer data, e.g.:
{code:java}
RUNTIME_PROFILE
aggregation(SUM, RowsReturned): 2142543
{code}
However, the counters are pretty-printed in the runtime profile, which means
that if they are greater than 1000, a shortened version is printed first, then
the accurate number comes in parenthesis , e.g.:
{code:java}
RowsReturned: 2.14M (2142543){code}
When the test framework parses the value of an aggregated counter, it wrongly
tries to parse the short version as a number, which returns a wrong value (2
instead of 2142543 in the example).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-8244) Toolchain build fails to publish binaries even if asked to do so
[
https://issues.apache.org/jira/browse/IMPALA-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Gaal resolved IMPALA-8244.
-
Resolution: Fixed
Fix Version/s: Impala 3.2.0
[https://github.com/cloudera/native-toolchain/commit/4e28c6c4ebf032a480549d73c4f93e733a65d075]
IMPALA-8244: Fix publishing for non-Docker builds
A previous commit introduced two additional environment variables to control
publishing: PUBLISH_DEPENDENCIES_S3 and PUBLISH_DEPENDENCIES_ARTIFACTORY.
These were set both from init.sh and from functions.sh. However, both init.sh
and functions.sh are sourced, not called, so by the time control reached
functions.sh, the variables were always defined, so the initial value of 0 could
never be overridden. This prevented the artifacts from being published even
if PUBLISH_DEPENDENCIES was set to 1 to request this.
This patch fixes the problem by removing the initialization and definition of
the
dependent variables from init.sh.
A seemingly benign typo is also removed from functions.sh
Tested by running a publishing build with the change using the regular
internal publishing job and observing the published results in the S3
bucket (and verifying build logs).
Change-Id: I4c8c6f4eeb7164e29ebc7c9b80c53da81537f63a
Reviewed-on: http://gerrit.cloudera.org:8080/12573
Reviewed-by: hector.aco...@cloudera.com
Reviewed-by: Tim Armstrong
Tested-by: Laszlo Gaal
master
> Toolchain build fails to publish binaries even if asked to do so
>
>
> Key: IMPALA-8244
> URL: https://issues.apache.org/jira/browse/IMPALA-8244
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Laszlo Gaal
>Assignee: Laszlo Gaal
>Priority: Critical
> Labels: native-toolchain, toolchain
> Fix For: Impala 3.2.0
>
>
> The recent commit to native-toolchain that enabled the publishing of binary
> artifacts from Docker-based toolchain builds
> ([https://github.com/cloudera/native-toolchain/commit/eeccbcab224fca5c25a7b78898444037b94d88b7)]
> changed the logic that gates the publishing calls.
> Internal builds on Jenkins (using the same logic as before the change) now
> don't even attempt publishing even if parameterized to do so. Excerpt from
> the build log:
> {code:java}
> 02:19:44.605 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.608 install -m 644 -p libcctz.a
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.860 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.863 install -m 644 -p libcctz.so.2.2
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.880 make: Leaving directory
> `/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
> 02:19:45.399 Cleaning
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
> ...
> 02:19:45.400 Entering directory
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
> 02:19:45.457 Removing cctz-2.2.tar.gz
> 02:19:45.457 Removing cctz-2.2/
> 02:19:45.459 Returning to directory
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain
> 02:19:45.459 # Build Complete cctz-2.2
> 02:19:45.459
> ###
> 02:19:45.549 + popd
> 02:19:45.549
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem
> 02:19:45.579 Archiving artifacts
> 02:19:45.781 [description-setter] Description set:
> ***redacted**
> 02:19:46.010 [WS-CLEANUP] Deleting project workspace...
> 02:19:46.010 [WS-CLEANUP] Deferred wipeout is used...
> 02:19:46.057 [WS-CLEANUP] done
> 02:19:46.057 Email was triggered for: Success
> 02:19:46.057 Sending email for trigger: Success
> 02:19:46.523 Sending email to: laszlo.g...@cloudera.com
> 02:19:46.551 Finished: SUCCESS{code}
> The happy case with the missing steps (copied from another job that published
> the bits successfully):
> {code:java}
> 02:26:22.083 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.085 install -m 644 -p
[jira] [Resolved] (IMPALA-8244) Toolchain build fails to publish binaries even if asked to do so
[
https://issues.apache.org/jira/browse/IMPALA-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Gaal resolved IMPALA-8244.
-
Resolution: Fixed
Fix Version/s: Impala 3.2.0
[https://github.com/cloudera/native-toolchain/commit/4e28c6c4ebf032a480549d73c4f93e733a65d075]
IMPALA-8244: Fix publishing for non-Docker builds
A previous commit introduced two additional environment variables to control
publishing: PUBLISH_DEPENDENCIES_S3 and PUBLISH_DEPENDENCIES_ARTIFACTORY.
These were set both from init.sh and from functions.sh. However, both init.sh
and functions.sh are sourced, not called, so by the time control reached
functions.sh, the variables were always defined, so the initial value of 0 could
never be overridden. This prevented the artifacts from being published even
if PUBLISH_DEPENDENCIES was set to 1 to request this.
This patch fixes the problem by removing the initialization and definition of
the
dependent variables from init.sh.
A seemingly benign typo is also removed from functions.sh
Tested by running a publishing build with the change using the regular
internal publishing job and observing the published results in the S3
bucket (and verifying build logs).
Change-Id: I4c8c6f4eeb7164e29ebc7c9b80c53da81537f63a
Reviewed-on: http://gerrit.cloudera.org:8080/12573
Reviewed-by: hector.aco...@cloudera.com
Reviewed-by: Tim Armstrong
Tested-by: Laszlo Gaal
master
> Toolchain build fails to publish binaries even if asked to do so
>
>
> Key: IMPALA-8244
> URL: https://issues.apache.org/jira/browse/IMPALA-8244
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
>Reporter: Laszlo Gaal
>Assignee: Laszlo Gaal
>Priority: Critical
> Labels: native-toolchain, toolchain
> Fix For: Impala 3.2.0
>
>
> The recent commit to native-toolchain that enabled the publishing of binary
> artifacts from Docker-based toolchain builds
> ([https://github.com/cloudera/native-toolchain/commit/eeccbcab224fca5c25a7b78898444037b94d88b7)]
> changed the logic that gates the publishing calls.
> Internal builds on Jenkins (using the same logic as before the change) now
> don't even attempt publishing even if parameterized to do so. Excerpt from
> the build log:
> {code:java}
> 02:19:44.605 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.608 install -m 644 -p libcctz.a
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.860 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.863 install -m 644 -p libcctz.so.2.2
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:19:44.880 make: Leaving directory
> `/data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz/cctz-2.2/build'
> 02:19:45.399 Cleaning
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
> ...
> 02:19:45.400 Entering directory
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/source/cctz
> 02:19:45.457 Removing cctz-2.2.tar.gz
> 02:19:45.457 Removing cctz-2.2/
> 02:19:45.459 Returning to directory
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain
> 02:19:45.459 # Build Complete cctz-2.2
> 02:19:45.459
> ###
> 02:19:45.549 + popd
> 02:19:45.549
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem
> 02:19:45.579 Archiving artifacts
> 02:19:45.781 [description-setter] Description set:
> ***redacted**
> 02:19:46.010 [WS-CLEANUP] Deleting project workspace...
> 02:19:46.010 [WS-CLEANUP] Deferred wipeout is used...
> 02:19:46.057 [WS-CLEANUP] done
> 02:19:46.057 Email was triggered for: Success
> 02:19:46.057 Sending email for trigger: Success
> 02:19:46.523 Sending email to: laszlo.g...@cloudera.com
> 02:19:46.551 Finished: SUCCESS{code}
> The happy case with the missing steps (copied from another job that published
> the bits successfully):
> {code:java}
> 02:26:22.083 install -d
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent58-ec2-c3-4xl-ondem/toolchain/build/cctz-2.2/lib
> 02:26:22.085 install -m 644 -p
[jira] [Updated] (IMPALA-4784) Remove InProcessStatestore
[
https://issues.apache.org/jira/browse/IMPALA-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-4784:
---
Fix Version/s: Impala 2.13.0
> Remove InProcessStatestore
> --
>
> Key: IMPALA-4784
> URL: https://issues.apache.org/jira/browse/IMPALA-4784
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend, Distributed Exec
>Affects Versions: Impala 2.9.0
>Reporter: Henry Robinson
>Assignee: Sailesh Mukil
>Priority: Major
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> The {{InProcessStatestore}} class is only used by the {{statestore-test}} and
> is likely to be obsoleted by the KRPC work. Since {{statestore-test}} doesn't
> provide much test coverage at all, let's remove {{InProcessStatestore}} to
> avoid the need to keep it up-to-date.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
