[jira] [Commented] (IMPALA-10196) Remove LlvmCodeGen::CastPtrToLlvmPtr
[ https://issues.apache.org/jira/browse/IMPALA-10196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203493#comment-17203493 ] Daniel Becker commented on IMPALA-10196: https://gerrit.cloudera.org/#/c/16517/ > Remove LlvmCodeGen::CastPtrToLlvmPtr > > > Key: IMPALA-10196 > URL: https://issues.apache.org/jira/browse/IMPALA-10196 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > LlvmCodeGen::CastPtrToLlvmPtr embeds a pointer that points to data in the > current process's memory into codegen'd IR code. Our long term goal is to > share the codegen'd IR among processes working on the same fragment, which is > not possible if the IR contains pointers pointing to data of a specific > process. A step in making the IR independent of the process generating it is > removing LlvmCodeGen::CastPtrToLlvmPtr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10196) Remove LlvmCodeGen::CastPtrToLlvmPtr
Daniel Becker created IMPALA-10196: -- Summary: Remove LlvmCodeGen::CastPtrToLlvmPtr Key: IMPALA-10196 URL: https://issues.apache.org/jira/browse/IMPALA-10196 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Daniel Becker Assignee: Daniel Becker LlvmCodeGen::CastPtrToLlvmPtr embeds a pointer that points to data in the current process's memory into codegen'd IR code. Our long term goal is to share the codegen'd IR among processes working on the same fragment, which is not possible if the IR contains pointers pointing to data of a specific process. A step in making the IR independent of the process generating it is removing LlvmCodeGen::CastPtrToLlvmPtr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10196) Remove LlvmCodeGen::CastPtrToLlvmPtr
Daniel Becker created IMPALA-10196: -- Summary: Remove LlvmCodeGen::CastPtrToLlvmPtr Key: IMPALA-10196 URL: https://issues.apache.org/jira/browse/IMPALA-10196 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Daniel Becker Assignee: Daniel Becker LlvmCodeGen::CastPtrToLlvmPtr embeds a pointer that points to data in the current process's memory into codegen'd IR code. Our long term goal is to share the codegen'd IR among processes working on the same fragment, which is not possible if the IR contains pointers pointing to data of a specific process. A step in making the IR independent of the process generating it is removing LlvmCodeGen::CastPtrToLlvmPtr. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMPALA-7656) Remove all uses of GetCodegendComputeFnWrapper()
[ https://issues.apache.org/jira/browse/IMPALA-7656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7656: -- Fix Version/s: Impala 4.0 > Remove all uses of GetCodegendComputeFnWrapper() > > > Key: IMPALA-7656 > URL: https://issues.apache.org/jira/browse/IMPALA-7656 > Project: IMPALA > Issue Type: Epic > Components: Backend >Reporter: Tim Armstrong >Assignee: Daniel Becker >Priority: Major > Labels: codegen > Fix For: Impala 4.0 > > > We should really get rid of all uses of this function, it was a stopgap to > add codegen support to expressions without really doing the work, but its > output can be 10x slower than doing it properly, e.g. see IMPALA-7655 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7656) Remove all uses of GetCodegendComputeFnWrapper()
[ https://issues.apache.org/jira/browse/IMPALA-7656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203475#comment-17203475 ] Tim Armstrong commented on IMPALA-7656: --- Woohoo! I'm so excited to see this closed. > Remove all uses of GetCodegendComputeFnWrapper() > > > Key: IMPALA-7656 > URL: https://issues.apache.org/jira/browse/IMPALA-7656 > Project: IMPALA > Issue Type: Epic > Components: Backend >Reporter: Tim Armstrong >Assignee: Daniel Becker >Priority: Major > Labels: codegen > Fix For: Impala 4.0 > > > We should really get rid of all uses of this function, it was a stopgap to > add codegen support to expressions without really doing the work, but its > output can be 10x slower than doing it properly, e.g. see IMPALA-7655 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10078) Proper codegen for KuduPartitionExpr
[ https://issues.apache.org/jira/browse/IMPALA-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-10078. Resolution: Implemented > Proper codegen for KuduPartitionExpr > > > Key: IMPALA-10078 > URL: https://issues.apache.org/jira/browse/IMPALA-10078 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Implement codegen for KuduPartitionExpr and remove the use of > GetCodegendComputeFnWrapper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-7656) Remove all uses of GetCodegendComputeFnWrapper()
[ https://issues.apache.org/jira/browse/IMPALA-7656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-7656. --- Resolution: Fixed > Remove all uses of GetCodegendComputeFnWrapper() > > > Key: IMPALA-7656 > URL: https://issues.apache.org/jira/browse/IMPALA-7656 > Project: IMPALA > Issue Type: Epic > Components: Backend >Reporter: Tim Armstrong >Assignee: Daniel Becker >Priority: Major > Labels: codegen > > We should really get rid of all uses of this function, it was a stopgap to > add codegen support to expressions without really doing the work, but its > output can be 10x slower than doing it properly, e.g. see IMPALA-7655 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7656) Remove all uses of GetCodegendComputeFnWrapper()
[ https://issues.apache.org/jira/browse/IMPALA-7656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-7656. --- Resolution: Fixed > Remove all uses of GetCodegendComputeFnWrapper() > > > Key: IMPALA-7656 > URL: https://issues.apache.org/jira/browse/IMPALA-7656 > Project: IMPALA > Issue Type: Epic > Components: Backend >Reporter: Tim Armstrong >Assignee: Daniel Becker >Priority: Major > Labels: codegen > > We should really get rid of all uses of this function, it was a stopgap to > add codegen support to expressions without really doing the work, but its > output can be 10x slower than doing it properly, e.g. see IMPALA-7655 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-10078) Proper codegen for KuduPartitionExpr
[ https://issues.apache.org/jira/browse/IMPALA-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-10078. Resolution: Implemented > Proper codegen for KuduPartitionExpr > > > Key: IMPALA-10078 > URL: https://issues.apache.org/jira/browse/IMPALA-10078 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Implement codegen for KuduPartitionExpr and remove the use of > GetCodegendComputeFnWrapper. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10175) Extend error message when cast(..format..) fails in parse phase
[
https://issues.apache.org/jira/browse/IMPALA-10175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203469#comment-17203469
]
ASF subversion and git services commented on IMPALA-10175:
--
Commit ee9904bc504edb19655bbee045c9b9e6a711eb7b in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee9904b ]
IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
The previous error message contained only the input string but
including the format string as well would make debugging easier.
Extended error message:
SELECT CAST('0;367' as date format 'YY;DDD');
String to Date parse failed. Input '0;367' doesn't match with
format 'YY;IDD'
Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
Reviewed-on: http://gerrit.cloudera.org:8080/16473
Tested-by: Impala Public Jenkins
Reviewed-by: Zoltan Borok-Nagy
> Extend error message when cast(..format..) fails in parse phase
> ---
>
> Key: IMPALA-10175
> URL: https://issues.apache.org/jira/browse/IMPALA-10175
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Gabor Kaszab
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: supportability
>
> {code:java}
> select cast('0;367' as date format 'YY;DDD');
> ERROR: UDF ERROR: String to Date parse failed. Invalid string val: "0;367"
> {code}
> Here the output contains the input string but would be more helpful for
> debugging if it also contained the original format string as well.
> This applies to String to Date conversions only as String to Timestamp
> failures currently doesn't raise an error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10112) Consider skipping FpRateTooHigh() check for bloom filters
[ https://issues.apache.org/jira/browse/IMPALA-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203468#comment-17203468 ] ASF subversion and git services commented on IMPALA-10112: -- Commit 5508be2b6273edd8074f466a7a64e70ecf304ddf in impala's branch refs/heads/master from Riza Suminto [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5508be2 ] IMPALA-10112: Remove FpRateTooHigh() check for bloom filter FpRateTooHigh() check for bloom filter that can be disabled if the observed false-positive probability (FPP) rate is higher than FLAGS_max_filter_error_rate. This patch remove FpRateTooHigh() check for several reasons: 1. Partition filters are probably still worth evaluating even if there are false positives because it is cheap and beneficial to eliminate a partition. 2. Runtime filters are dynamically disabled on the scan side if they are ineffective. Even an ALWAYS_TRUE_FILTER that substitutes a disabled filter is also still being evaluated and not entirely free. 3. The disabling is less likely to kick in for partitioned joins because it only applied to a small subset of the filters before the Or() operation. 4. FpRateTooHigh() use num_build_rows to approximate the FPP rate of the resulting filter. This can be inaccurate because it does not take account of duplicate values of the filter key on the build side. This patch also removes some tests in test_runtime_filters.py that check cancellation of filters having a high FPP rate. Testing: - Run and pass core tests. - Manually test and verify in a real large cluster (TPC-DS 30TB scale) that there is only a little (< 2.1%) to no performance regression incurred from the removal of high FPP check. The test used TPC-DS queries Q14a, Q50, Q64, Q71, Q84, Q93, and a modified Q93 with the left outer join replaced by an inner join. The query time comparison between having an FPP check versus without having an FPP check are the following: | Query | With check (ms) | Without check (ms) | |---+-+| | Q14a | 129801 | 125992 | | Q50 | 72519 | 72652 | | Q64 | 150684 | 145241 | | Q71 | 21009 | 20358 | | Q84 | 29334 | 29944 | | Q93 | 91782 | 91923 | | Q93m | 92897 | 92135 | Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4 Reviewed-on: http://gerrit.cloudera.org:8080/16499 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Consider skipping FpRateTooHigh() check for bloom filters > - > > Key: IMPALA-10112 > URL: https://issues.apache.org/jira/browse/IMPALA-10112 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Assignee: Riza Suminto >Priority: Major > Labels: performance > Fix For: Impala 4.0 > > > This check disables bloom filters on the sender side. > It is inaccurate in cases where there are duplicate values of the filter key > on the build side. E.g. many-to-many join or a join with multiple keys. This > could be fixed with some effort, but is probably not worth it, because: > * Partition filters are probably still worth evaluating even if there are > false positives, because it's cheap and eliminating a partition is still > beneficial. > * Runtime filters are dynamically disabled on the scan side if they are > ineffective. I think we still also "evaluate" the always true filter, which > is cheaper than doing the hashing and bloom evaluation, but still not > entirely free. > * The disabling is fairly unlikely to kick in for partitioned joins because > it's only applied to a small subset of the filter, before the Or() operation. > So it's potentially harmful and only likely beneficial for broadcast join > filters, in which case it saves a small amount of scan CPU and, for global > filters, coordinator RPCs and broadcasting. It's unclear that the complexity > is worth it for this relatively small and uncertain benefit. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10078) Proper codegen for KuduPartitionExpr
[ https://issues.apache.org/jira/browse/IMPALA-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203467#comment-17203467 ] ASF subversion and git services commented on IMPALA-10078: -- Commit 5d9daf40511e8b43d9bfd186b19efc06588cd1ce in impala's branch refs/heads/master from Daniel Becker [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5d9daf4 ] IMPALA-10078: Proper codegen for KuduPartitionExpr Implementing codegen for KuduPartitionExpr. Testing: - Added a sanity test that checks that numbers are evenly distributed when inserted into a Kudu table. Change-Id: Ifcae34f71b407837e2c5f1b97aa230e490a268df Reviewed-on: http://gerrit.cloudera.org:8080/16419 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Proper codegen for KuduPartitionExpr > > > Key: IMPALA-10078 > URL: https://issues.apache.org/jira/browse/IMPALA-10078 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Implement codegen for KuduPartitionExpr and remove the use of > GetCodegendComputeFnWrapper. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8577) Crash during OpenSSLSocket.read
[
https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved IMPALA-8577.
--
Fix Version/s: Impala 4.0
Resolution: Fixed
This was fixed a while ago. Impala has been using wildfly for communication
with S3 for a while now and everything seems stable.
> Crash during OpenSSLSocket.read
> ---
>
> Key: IMPALA-8577
> URL: https://issues.apache.org/jira/browse/IMPALA-8577
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: David Rorke
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Impala 4.0
>
> Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp,
> hs_err_pid6313.log,
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313,
>
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313
>
>
> Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from
> the stack trace (hs_err log file attached with more complete stack):
> {noformat}
> Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free
> space=8180k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [impalad+0x2528a33]
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int)+0x133
> C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f
> C [impalad+0x266468a] operator delete(void*)+0x32a
> C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d
> J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @
> 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9]
> J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @
> 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c]
> J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @
> 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c]
> J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @
> 0x7f3d4de419cc [0x7f3d4de417c0+0x20c]
> J 2994 C2
> org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
> (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134]
> J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @
> 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0]
> J 5090 C2
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I
> (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0]
> J 5846 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I
> (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384]
> J 5845 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31
> bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c]
> {noformat}
> The crash may not be easy to reproduce. I've run this test multiple times
> and only crashed once. I have a core file if needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8577) Crash during OpenSSLSocket.read
[
https://issues.apache.org/jira/browse/IMPALA-8577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved IMPALA-8577.
--
Fix Version/s: Impala 4.0
Resolution: Fixed
This was fixed a while ago. Impala has been using wildfly for communication
with S3 for a while now and everything seems stable.
> Crash during OpenSSLSocket.read
> ---
>
> Key: IMPALA-8577
> URL: https://issues.apache.org/jira/browse/IMPALA-8577
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: David Rorke
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Impala 4.0
>
> Attachments: 5ca78771-ad78-4a29-31f88aa6-9bfac38c.dmp,
> hs_err_pid6313.log,
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.ERROR.20190521-103105.6313,
>
> impalad.drorke-impala-r5d2xl2-30w-17.vpc.cloudera.com.impala.log.INFO.20190521-103105.6313
>
>
> Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from
> the stack trace (hs_err log file attached with more complete stack):
> {noformat}
> Stack: [0x7f3d095bc000,0x7f3d09dbc000], sp=0x7f3d09db9050, free
> space=8180k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [impalad+0x2528a33]
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int)+0x133
> C [impalad+0x2528e0f] tcmalloc::ThreadCache::Scavenge()+0x3f
> C [impalad+0x266468a] operator delete(void*)+0x32a
> C [libcrypto.so.10+0x6e70d] CRYPTO_free+0x1d
> J 5709 org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @
> 0x7f3d4dadf9f9 [0x7f3d4dadf940+0xb9]
> J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @
> 0x7f3d4dfd0dfc [0x7f3d4dfd0d80+0x7c]
> J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @
> 0x7f3d4de4fe2c [0x7f3d4de4f720+0x70c]
> J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @
> 0x7f3d4de419cc [0x7f3d4de417c0+0x20c]
> J 2994 C2
> org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
> (892 bytes) @ 0x7f3d4db8da34 [0x7f3d4db8c900+0x1134]
> J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @
> 0x7f3d4dd64cb0 [0x7f3d4dd646c0+0x5f0]
> J 5090 C2
> com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I
> (97 bytes) @ 0x7f3d4ddd9ee0 [0x7f3d4ddd9e40+0xa0]
> J 5846 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I
> (48 bytes) @ 0x7f3d4d7acb24 [0x7f3d4d7ac7a0+0x384]
> J 5845 C1
> com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31
> bytes) @ 0x7f3d4d7ad49c [0x7f3d4d7ad220+0x27c]
> {noformat}
> The crash may not be easy to reproduce. I've run this test multiple times
> and only crashed once. I have a core file if needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMPALA-941) Impala Parser issue when using fully qualified table names that start with a number.
[
https://issues.apache.org/jira/browse/IMPALA-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203436#comment-17203436
]
Tim Armstrong commented on IMPALA-941:
--
[~Sethi] looks like we didn't mark it as done in help wanted. We have lots more
newbie JIRAs if you want to make a contribution though - see
https://issues.apache.org/jira/issues/?jql=project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20labels%20%3D%20newbie
Or if you have an area you want to learn about, reach out to
d...@impala.apache.org and we can probably find a suitable JIRA for an
improvement (there's lots of things we want to do that we don't have enough
contributors to do)!
> Impala Parser issue when using fully qualified table names that start with a
> number.
>
>
> Key: IMPALA-941
> URL: https://issues.apache.org/jira/browse/IMPALA-941
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 1.2.4
>Reporter: Udai Kiran Potluri
>Assignee: Fredy Wijaya
>Priority: Minor
> Labels: customer, newbie
> Fix For: Impala 3.2.0
>
>
> Impala Parser issue when using fully qualified table names that start with a
> number.
> What is going wrong is that Impala is parsing the table name as a floating
> point literal since the text starts with a number and contains a period.
> {code}
> INVALIDATE METADATA db.571_market;
> Query: invalidate METADATA db.571_market
> ERROR: AnalysisException: Syntax error in line 1:
> invalidate METADATA db.571_market
> ^
> Encountered: FLOATING POINT LITERAL
> Expected: ADD, AS, CHANGE, COMMENT, CROSS, DROP, FULL, GROUP, HAVING, INNER,
> JOIN, LEFT, LIKE, LIMIT, LOCATION, ON, ORDER, PARTITION, RENAME, REPLACE,
> RIGHT, ROW, SELECT, SET, STORED, TBLPROPERTIES, UNION, USING, VALUES, WHERE,
> WITH, COMMA, IDENTIFIER
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10193) Limit the memory usage of the whole mini-cluster
[ https://issues.apache.org/jira/browse/IMPALA-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203335#comment-17203335 ] Tim Armstrong commented on IMPALA-10193: [~fifteencai] are you intending to work on this yourself? I just added you as a contributor to the IMPALA JIRA project so you can assign it to yourself. Maybe we can leverage IMPALA-7940? Impala daemons can already detect the docker memory limit. If mem_limit is set to a % value, then the impalad mem_limit will be that % of the container memory limit. > Limit the memory usage of the whole mini-cluster > > > Key: IMPALA-10193 > URL: https://issues.apache.org/jira/browse/IMPALA-10193 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: Fifteen >Priority: Minor > Attachments: image-2020-09-28-17-18-15-358.png > > > The mini-cluster contains 3 virtual nodes, and all of them runs in a single > 'Machine'. By quoting, it implies the machine can be a docker container. If > the container is started with `-priviledged` and the actual memory is limited > by CGROUPS, then the total memory in `top` and the actual available memory > can be different! > > For example, in the container below, `top` tells us the total memory is > 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal > mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get > killed. > !image-2020-09-28-17-18-15-358.png! > > So we may need a way to limit the whole mini-cluster's memory usage. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10195) TestAdmissionControllerStress test_mem_limit run_admission_test looks flaky again
[
https://issues.apache.org/jira/browse/IMPALA-10195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Gaal reassigned IMPALA-10195:
Assignee: Bikramjeet Vig
> TestAdmissionControllerStress test_mem_limit run_admission_test looks flaky
> again
> -
>
> Key: IMPALA-10195
> URL: https://issues.apache.org/jira/browse/IMPALA-10195
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Laszlo Gaal
>Assignee: Bikramjeet Vig
>Priority: Blocker
>
> custom_cluster.test_admission_controller.TestAdmissionControllerStress.test_mem_limit
> fails with a familiar assertion:
> {code}
> custom_cluster/test_admission_controller.py:1856: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1754: in run_admission_test
> assert metric_deltas['admitted'] >= expected_admitted
> E assert 0 >= 5
> {code}
> The failure looks eerily similar to IMPALA-8342. That one was closed 18
> months ago, so I'm hesitant to reopen it, but it is linked as related.
> Tagging [~bikramjeet.vig] and [~tarmstr...@cloudera.com] as the they looked
> at the same flakiness at that time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10195) TestAdmissionControllerStress test_mem_limit run_admission_test looks flaky again
Laszlo Gaal created IMPALA-10195:
Summary: TestAdmissionControllerStress test_mem_limit
run_admission_test looks flaky again
Key: IMPALA-10195
URL: https://issues.apache.org/jira/browse/IMPALA-10195
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.0
Reporter: Laszlo Gaal
custom_cluster.test_admission_controller.TestAdmissionControllerStress.test_mem_limit
fails with a familiar assertion:
{code}
custom_cluster/test_admission_controller.py:1856: in test_mem_limit
{'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
custom_cluster/test_admission_controller.py:1754: in run_admission_test
assert metric_deltas['admitted'] >= expected_admitted
E assert 0 >= 5
{code}
The failure looks eerily similar to IMPALA-8342. That one was closed 18 months
ago, so I'm hesitant to reopen it, but it is linked as related.
Tagging [~bikramjeet.vig] and [~tarmstr...@cloudera.com] as the they looked at
the same flakiness at that time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10195) TestAdmissionControllerStress test_mem_limit run_admission_test looks flaky again
Laszlo Gaal created IMPALA-10195:
Summary: TestAdmissionControllerStress test_mem_limit
run_admission_test looks flaky again
Key: IMPALA-10195
URL: https://issues.apache.org/jira/browse/IMPALA-10195
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.0
Reporter: Laszlo Gaal
custom_cluster.test_admission_controller.TestAdmissionControllerStress.test_mem_limit
fails with a familiar assertion:
{code}
custom_cluster/test_admission_controller.py:1856: in test_mem_limit
{'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
custom_cluster/test_admission_controller.py:1754: in run_admission_test
assert metric_deltas['admitted'] >= expected_admitted
E assert 0 >= 5
{code}
The failure looks eerily similar to IMPALA-8342. That one was closed 18 months
ago, so I'm hesitant to reopen it, but it is linked as related.
Tagging [~bikramjeet.vig] and [~tarmstr...@cloudera.com] as the they looked at
the same flakiness at that time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IMPALA-10194) ASAN use-after-free in hdfs-util-test
Laszlo Gaal created IMPALA-10194:
Summary: ASAN use-after-free in hdfs-util-test
Key: IMPALA-10194
URL: https://issues.apache.org/jira/browse/IMPALA-10194
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 4.0
Reporter: Laszlo Gaal
This is the ASAN report section from LastTest.log:
{code}
90/124 Testing: hdfs-util-test
90/124 Test: hdfs-util-test
Command:
"/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/run-jvm-binary.sh"
"/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug//util/hdfs-util-test"
"-log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/be_tests"
Directory:
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
"hdfs-util-test" start time: Sep 25 22:18 PDT
Output:
--
seed = 1601097520
Note: Google Test filter = HdfsUtilTest.*
[==] Running 2 tests from 1 test case.
[--] Global test environment set-up.
[--] 2 tests from HdfsUtilTest
[ RUN ] HdfsUtilTest.CheckFilesystemsMatch
20/09/25 22:18:40 INFO util.JvmPauseMonitor: Starting JVM pause monitor
[ OK ] HdfsUtilTest.CheckFilesystemsMatch (941 ms)
[ RUN ] HdfsUtilTest.CheckGetBaseName
[ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
[--] 2 tests from HdfsUtilTest (941 ms total)
[--] Global test environment tear-down
[==] 2 tests from 1 test case ran. (941 ms total)
[ PASSED ] 2 tests.
YOU HAVE 1 DISABLED TEST
=
==17289==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070e440
at pc 0x01e3854b bp 0x7effcdf609e0 sp 0x7effcdf60190
READ of size 7 at 0x6070e440 thread T4
Test time = 2.13 sec
--
Test Passed.
"hdfs-util-test" end time: Sep 25 22:18 PDT
"hdfs-util-test" time elapsed: 00:00:02
--
{code}
The following two commits have been added to master since the last green ASAN
run:
{code}
IMPALA-10183: Fix hitting DCHECK when cancelling a query with result spooling
IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IMPALA-10194) ASAN use-after-free in hdfs-util-test
Laszlo Gaal created IMPALA-10194:
Summary: ASAN use-after-free in hdfs-util-test
Key: IMPALA-10194
URL: https://issues.apache.org/jira/browse/IMPALA-10194
Project: IMPALA
Issue Type: Bug
Affects Versions: Impala 4.0
Reporter: Laszlo Gaal
This is the ASAN report section from LastTest.log:
{code}
90/124 Testing: hdfs-util-test
90/124 Test: hdfs-util-test
Command:
"/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/run-jvm-binary.sh"
"/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug//util/hdfs-util-test"
"-log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/be_tests"
Directory:
/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util
"hdfs-util-test" start time: Sep 25 22:18 PDT
Output:
--
seed = 1601097520
Note: Google Test filter = HdfsUtilTest.*
[==] Running 2 tests from 1 test case.
[--] Global test environment set-up.
[--] 2 tests from HdfsUtilTest
[ RUN ] HdfsUtilTest.CheckFilesystemsMatch
20/09/25 22:18:40 INFO util.JvmPauseMonitor: Starting JVM pause monitor
[ OK ] HdfsUtilTest.CheckFilesystemsMatch (941 ms)
[ RUN ] HdfsUtilTest.CheckGetBaseName
[ OK ] HdfsUtilTest.CheckGetBaseName (0 ms)
[--] 2 tests from HdfsUtilTest (941 ms total)
[--] Global test environment tear-down
[==] 2 tests from 1 test case ran. (941 ms total)
[ PASSED ] 2 tests.
YOU HAVE 1 DISABLED TEST
=
==17289==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070e440
at pc 0x01e3854b bp 0x7effcdf609e0 sp 0x7effcdf60190
READ of size 7 at 0x6070e440 thread T4
Test time = 2.13 sec
--
Test Passed.
"hdfs-util-test" end time: Sep 25 22:18 PDT
"hdfs-util-test" time elapsed: 00:00:02
--
{code}
The following two commits have been added to master since the last green ASAN
run:
{code}
IMPALA-10183: Fix hitting DCHECK when cancelling a query with result spooling
IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10193) Limit the memory usage of the whole mini-cluster
[ https://issues.apache.org/jira/browse/IMPALA-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fifteen updated IMPALA-10193: - Attachment: image-2020-09-28-17-18-15-358.png > Limit the memory usage of the whole mini-cluster > > > Key: IMPALA-10193 > URL: https://issues.apache.org/jira/browse/IMPALA-10193 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: Fifteen >Priority: Minor > Attachments: image-2020-09-28-17-18-15-358.png > > > The mini-cluster contains 3 virtual nodes, and all of them runs in a single > 'Machine'. By quoting, it implies the machine can be a docker container. If > the container is started with `-priviledged` and the actual memory is limited > by CGROUPS, then the total memory in `top` and the actual available memory > can be different! > > For example, in the container below, `top` tells us the total memory is > 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal > mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get > killed. > !image-2020-09-28-17-08-02-604.png! > > So we may need a way to limit the whole mini-cluster's memory usage. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10193) Limit the memory usage of the whole mini-cluster
[ https://issues.apache.org/jira/browse/IMPALA-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fifteen updated IMPALA-10193: - Description: The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `top` and the actual available memory can be different! For example, in the container below, `top` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed. !image-2020-09-28-17-18-15-358.png! So we may need a way to limit the whole mini-cluster's memory usage. was: The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `top` and the actual available memory can be different! For example, in the container below, `top` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed. !image-2020-09-28-17-08-02-604.png! So we may need a way to limit the whole mini-cluster's memory usage. > Limit the memory usage of the whole mini-cluster > > > Key: IMPALA-10193 > URL: https://issues.apache.org/jira/browse/IMPALA-10193 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.4.0 >Reporter: Fifteen >Priority: Minor > Attachments: image-2020-09-28-17-18-15-358.png > > > The mini-cluster contains 3 virtual nodes, and all of them runs in a single > 'Machine'. By quoting, it implies the machine can be a docker container. If > the container is started with `-priviledged` and the actual memory is limited > by CGROUPS, then the total memory in `top` and the actual available memory > can be different! > > For example, in the container below, `top` tells us the total memory is > 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal > mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get > killed. > !image-2020-09-28-17-18-15-358.png! > > So we may need a way to limit the whole mini-cluster's memory usage. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10193) Limit the memory usage of the whole mini-cluster
Fifteen created IMPALA-10193: Summary: Limit the memory usage of the whole mini-cluster Key: IMPALA-10193 URL: https://issues.apache.org/jira/browse/IMPALA-10193 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 3.4.0 Reporter: Fifteen The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `top` and the actual available memory can be different! For example, in the container below, `top` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed. !image-2020-09-28-17-08-02-604.png! So we may need a way to limit the whole mini-cluster's memory usage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10193) Limit the memory usage of the whole mini-cluster
Fifteen created IMPALA-10193: Summary: Limit the memory usage of the whole mini-cluster Key: IMPALA-10193 URL: https://issues.apache.org/jira/browse/IMPALA-10193 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 3.4.0 Reporter: Fifteen The mini-cluster contains 3 virtual nodes, and all of them runs in a single 'Machine'. By quoting, it implies the machine can be a docker container. If the container is started with `-priviledged` and the actual memory is limited by CGROUPS, then the total memory in `top` and the actual available memory can be different! For example, in the container below, `top` tells us the total memory is 128GB, while the total memory set in CGROUPS is actually 32GB. If the acutal mem usage exceeds 32GB, process (such as impalad, hivemaster2 etc.) get killed. !image-2020-09-28-17-08-02-604.png! So we may need a way to limit the whole mini-cluster's memory usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
