[jira] [Updated] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7222: Reviewer: Anton Gozhiy (was: Aman Sinha) > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Gozhiy updated DRILL-7222: Labels: doc-impacting user-experience (was: doc-impacting ready-to-commit user-experience) > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913242#comment-16913242 ] ASF GitHub Bot commented on DRILL-7222: --- agozhiy commented on issue #1779: DRILL-7222: Visualize estimated and actual row counts for a query URL: https://github.com/apache/drill/pull/1779#issuecomment-523866520 @kkhatua, please address one more comment and squash the commits. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zinoviev updated DRILL-7355: - Summary: Reading a offset based vectors may produce IndexOutOfBoundsException (was: Reading a offset based vectors into holder may produce IndexOutOfBoundsException) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. When reading values into the corresponding holder, the values of > their offset vectors are written into the "start" and "end" fields as > ordinary integers. In the subsequent operations is not considered that > originally this values were unsigned integers and (for example in the > calculation of the buffer size at ParquetOutputRecordWriter) usual > subtraction "end" - "start" used. This causes the target buffer size to be > negative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (DRILL-7355) Reading a offset based vectors into holder may produce IndexOutOfBoundsException
Oleg Zinoviev created DRILL-7355: Summary: Reading a offset based vectors into holder may produce IndexOutOfBoundsException Key: DRILL-7355 URL: https://issues.apache.org/jira/browse/DRILL-7355 Project: Apache Drill Issue Type: Bug Affects Versions: 1.16.0 Environment: Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) Reporter: Oleg Zinoviev Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset vector. When reading values into the corresponding holder, the values of their offset vectors are written into the "start" and "end" fields as ordinary integers. In the subsequent operations is not considered that originally this values were unsigned integers and (for example in the calculation of the buffer size at ParquetOutputRecordWriter) usual subtraction "end" - "start" used. This causes the target buffer size to be negative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913276#comment-16913276 ] Oleg Zinoviev commented on DRILL-7355: -- Any chance this can be fixed in 1.17.0? > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. When reading values into the corresponding holder, the values of > their offset vectors are written into the "start" and "end" fields as > ordinary integers. In the subsequent operations is not considered that > originally this values were unsigned integers and (for example in the > calculation of the buffer size at ParquetOutputRecordWriter) usual > subtraction "end" - "start" used. This causes the target buffer size to be > negative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7355:
Environment: (was: Stack trace:
Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
(expected: range(0, 524288))
at
io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
at
io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
at
org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
at
org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283))
> Reading a offset based vectors may produce IndexOutOfBoundsException
>
>
> Key: DRILL-7355
> URL: https://issues.apache.org/jira/browse/DRILL-7355
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Priority: Major
>
> Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset
> vector. When reading values into the corresponding holder, the values of
> their offset vectors are written into the "start" and "end" fields as
> ordinary integers. In the subsequent operations is not considered that
> originally this values were unsigned integers and (for example in the
> calculation of the buffer size at ParquetOutputRecordWriter) usual
> subtraction "end" - "start" used. This causes the target buffer size to be
> negative.
> Stack trace:
> {noformat}
> Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
> (expected: range(0, 524288))
> at
> io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
> at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
> at
> io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
> at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
> at
> io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
> at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
> at
> org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
> at
> org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
> at
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7355:
Description:
Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset
vector. When reading values into the corresponding holder, the values of their
offset vectors are written into the "start" and "end" fields as ordinary
integers. In the subsequent operations is not considered that originally this
values were unsigned integers and (for example in the calculation of the buffer
size at ParquetOutputRecordWriter) usual subtraction "end" - "start" used. This
causes the target buffer size to be negative.
Stack trace:
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
(expected: range(0, 524288))
at
io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
at
io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
at
org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
at
org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
{noformat}
was:Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4
offset vector. When reading values into the corresponding holder, the values of
their offset vectors are written into the "start" and "end" fields as ordinary
integers. In the subsequent operations is not considered that originally this
values were unsigned integers and (for example in the calculation of the buffer
size at ParquetOutputRecordWriter) usual subtraction "end" - "start" used. This
causes the target buffer size to be negative.
> Reading a offset based vectors may produce IndexOutOfBoundsException
>
>
> Key: DRILL-7355
> URL: https://issues.apache.org/jira/browse/DRILL-7355
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
> Environment: Stack trace:
> Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
> (expected: range(0, 524288))
> at
> io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
> at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
> at
> io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
> at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
> at
> io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
> at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
> at
> org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
> at
> org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
> at
> org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
> at
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
>Reporter: Oleg Zinoviev
>Priority: Major
>
> Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset
> vector. When reading values into the correspondi
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zinoviev updated DRILL-7355:
-
Description: * to be fixed * (was: Var16Char, VarBinary, VarChar and
VarDecimal vectors contain a UInt4 offset vector. When reading values into the
corresponding holder, the values of their offset vectors are written into the
"start" and "end" fields as ordinary integers. In the subsequent operations is
not considered that originally this values were unsigned integers and (for
example in the calculation of the buffer size at ParquetOutputRecordWriter)
usual subtraction "end" - "start" used. This causes the target buffer size to
be negative.
Stack trace:
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
(expected: range(0, 524288))
at
io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
at
io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
at
org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
at
org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
{noformat})
Environment:
Stack trace:
Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
(expected: range(0, 524288))
at
io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
at
io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
at
io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
at
org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824)
at
org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48)
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)
> Reading a offset based vectors may produce IndexOutOfBoundsException
>
>
> Key: DRILL-7355
> URL: https://issues.apache.org/jira/browse/DRILL-7355
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
> Environment: Stack trace:
> Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799
> (expected: range(0, 524288))
> at
> io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125)
> at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120)
> at
> io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244)
> at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734)
> at
> io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34)
> at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364)
> at
> org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConve
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913283#comment-16913283 ] Arina Ielchiieva commented on DRILL-7355: - [~le.louch] please add data file and example of the select statement so if any volunteers to fix this could be able to reproduce the issue. Regarding the fix availability in 1.17, contributions always welcome :) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7355: Description: * to be fixed * Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) was:* to be fixed * > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913284#comment-16913284 ] Oleg Zinoviev commented on DRILL-7355: -- Sorry, I missed that UInt4 is a Java 'int' vector. Description need to be fixed > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 > Environment: Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7355: Environment: (was: Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283)) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > * to be fixed * -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zinoviev updated DRILL-7355: - Description: Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset vector. When writing and reading these vectors, the overflow of the offset value is not processed. While reading values into the corresponding holder, the values of their offset vectors are written into the "start" and "end" fields as ordinary integers. w In some cases, the "start" and "end" values are used to get the size of the buffer (for example,ParquetOutputRecordWriter). This causes the target buffer size to be negative. Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) was: * to be fixed * Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. When writing and reading these vectors, the overflow of the offset > value is not processed. While reading values into the corresponding holder, > the values of their offset vectors are written into the "start" and "end" > fields as ordinary integers. w In some cases, the "start" and "end" values > are used to get the size of the buffer (for > example,ParquetOutputRecordWriter). This causes the target buffer size to be > negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(Abstr
[jira] [Updated] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zinoviev updated DRILL-7355: - Description: Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset vector. During the writing of these vectors, the offset value overflow is not processed. Therefore, upon subsequent reading of such a vector to the corresponding holder, the value in the "end" field may be less than the value of the "start" field. In some cases, the "start" and "end" values are used to get the size of the buffer (for example,ParquetOutputRecordWriter). This causes the target buffer size to be negative. Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) was: Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset vector. When writing and reading these vectors, the overflow of the offset value is not processed. While reading values into the corresponding holder, the values of their offset vectors are written into the "start" and "end" fields as ordinary integers. w In some cases, the "start" and "end" values are used to get the size of the buffer (for example,ParquetOutputRecordWriter). This causes the target buffer size to be negative. Stack trace: Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 (expected: range(0, 524288)) at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) at io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) at io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) at org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) at org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector
[jira] [Created] (DRILL-7356) Introduce new session option for enabling metastore
Volodymyr Vysotskyi created DRILL-7356:
--
Summary: Introduce new session option for enabling metastore
Key: DRILL-7356
URL: https://issues.apache.org/jira/browse/DRILL-7356
Project: Apache Drill
Issue Type: Task
Affects Versions: 1.17.0
Reporter: Volodymyr Vysotskyi
Assignee: Volodymyr Vysotskyi
Fix For: 1.17.0
Introduce a new session option for enabling metastore: {{metastore.enabled}}.
It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce new session option for enabling metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7356:
---
Reviewer: Arina Ielchiieva
> Introduce new session option for enabling metastore
> ---
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce session option for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7356:
Summary: Introduce session option for the Drill Metastore (was: Introduce
new session option for enabling metastore)
> Introduce session option for the Drill Metastore
>
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7356:
Summary: Introduce session options for the Drill Metastore (was: Introduce
session option for the Drill Metastore)
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi updated DRILL-7356:
---
Parent: DRILL-6552
Issue Type: Sub-task (was: Task)
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (DRILL-7357) Expose Drill Metastore Metadata through INFORMATION_SCHEMA
Arina Ielchiieva created DRILL-7357: --- Summary: Expose Drill Metastore Metadata through INFORMATION_SCHEMA Key: DRILL-7357 URL: https://issues.apache.org/jira/browse/DRILL-7357 Project: Apache Drill Issue Type: Sub-task Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.17.0 Document: https://docs.google.com/document/d/10CkLdrlUJUNRrHKLeo8jTUJB8xAP1D0byTOvn8wNoF0/edit#heading=h.gzj2dj5a4yds Sections: 5.19 INFORMATION_SCHEMA updates 4.3.2 Using the statistics -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7357) Expose Drill Metastore metadata through INFORMATION_SCHEMA
[ https://issues.apache.org/jira/browse/DRILL-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7357: Summary: Expose Drill Metastore metadata through INFORMATION_SCHEMA (was: Expose Drill Metastore Metadata through INFORMATION_SCHEMA) > Expose Drill Metastore metadata through INFORMATION_SCHEMA > -- > > Key: DRILL-7357 > URL: https://issues.apache.org/jira/browse/DRILL-7357 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.17.0 > > > Document: > https://docs.google.com/document/d/10CkLdrlUJUNRrHKLeo8jTUJB8xAP1D0byTOvn8wNoF0/edit#heading=h.gzj2dj5a4yds > Sections: > 5.19 INFORMATION_SCHEMA updates > 4.3.2 Using the statistics -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-6552) Drill Metadata management "Drill Metastore"
[ https://issues.apache.org/jira/browse/DRILL-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6552: Summary: Drill Metadata management "Drill Metastore" (was: Drill Metadata management "Drill MetaStore") > Drill Metadata management "Drill Metastore" > --- > > Key: DRILL-6552 > URL: https://issues.apache.org/jira/browse/DRILL-6552 > Project: Apache Drill > Issue Type: New Feature > Components: Metadata >Affects Versions: 1.13.0 >Reporter: Vitalii Diravka >Assignee: Volodymyr Vysotskyi >Priority: Major > Fix For: 1.17.0 > > > It would be useful for Drill to have some sort of metastore which would > enable Drill to remember previously defined schemata so Drill doesn’t have to > do the same work over and over again. > It allows to store schema and statistics, which will allow to accelerate > queries validation, planning and execution time. Also it increases stability > of Drill and allows to avoid different kind if issues: "schema change > Exceptions", "limit 0" optimization and so on. > One of the main candidates is Hive Metastore. > Starting from 3.0 version Hive Metastore can be the separate service from > Hive server: > [https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+3.0+Administration] > Optional enhancement is storing Drill's profiles, UDFs, plugins configs in > some kind of metastore as well. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913379#comment-16913379 ] Oleg Zinoviev commented on DRILL-7355: -- [~arina], looks like it's already fixed, since I cannot reproduce it on drill 1.17.0. > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector to the > corresponding holder, the value in the "end" field may be less than the value > of the "start" field. In some cases, the "start" and "end" values are used to > get the size of the buffer (for example,ParquetOutputRecordWriter). This > causes the target buffer size to be negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zinoviev closed DRILL-7355. Resolution: Cannot Reproduce > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector to the > corresponding holder, the value in the "end" field may be less than the value > of the "start" field. In some cases, the "start" and "end" values are used to > get the size of the buffer (for example,ParquetOutputRecordWriter). This > causes the target buffer size to be negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7355) Reading a offset based vectors may produce IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/DRILL-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913381#comment-16913381 ] Arina Ielchiieva commented on DRILL-7355: - Awesome :) > Reading a offset based vectors may produce IndexOutOfBoundsException > > > Key: DRILL-7355 > URL: https://issues.apache.org/jira/browse/DRILL-7355 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > > Var16Char, VarBinary, VarChar and VarDecimal vectors contain a UInt4 offset > vector. During the writing of these vectors, the offset value overflow is not > processed. Therefore, upon subsequent reading of such a vector to the > corresponding holder, the value in the "end" field may be less than the value > of the "start" field. In some cases, the "start" and "end" values are used to > get the size of the buffer (for example,ParquetOutputRecordWriter). This > causes the target buffer size to be negative. > Stack trace: > Caused by: java.lang.IndexOutOfBoundsException: index: 17, length: -799 > (expected: range(0, 524288)) > at > io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1125) > at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1120) > at > io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:244) > at io.netty.buffer.WrappedByteBuf.nioBuffer(WrappedByteBuf.java:734) > at > io.netty.buffer.UnsafeDirectLittleEndian.nioBuffer(UnsafeDirectLittleEndian.java:34) > at io.netty.buffer.DrillBuf.nioBuffer(DrillBuf.java:364) > at > org.apache.drill.exec.store.ParquetOutputRecordWriter$VarCharParquetConverter.writeField(ParquetOutputRecordWriter.java:1824) > at > org.apache.drill.exec.store.EventBasedRecordWriter.write(EventBasedRecordWriter.java:48) > at > org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:111) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) > at > org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:93) > at > org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:296) > at > org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:283) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (DRILL-7168) Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
[
https://issues.apache.org/jira/browse/DRILL-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-7168:
---
Assignee: Arina Ielchiieva
> Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
> --
>
> Key: DRILL-7168
> URL: https://issues.apache.org/jira/browse/DRILL-7168
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> By [~Paul.Rogers]:
> {quote}
> Sooner or later users are going to ask for a command to update just the
> properties, or just add or remove a column, without having to spell out the
> entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
> {quote}
> Syntax:
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> add
> [columns (col1 int, col2 varchar)]
> [properties ('prop1'='val1', 'prop2'='val2')]
> {code}
> Add command will add or replace existing columns or properties.
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> remove
> [columns (`col1`, `col2`)]
> [properties ('prop1', 'prop2')]
> {code}
> Remove command won't fail if column or property does not exist, will return
> false status only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7168) Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
[
https://issues.apache.org/jira/browse/DRILL-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7168:
Description:
By [~Paul.Rogers]:
{quote}
Sooner or later users are going to ask for a command to update just the
properties, or just add or remove a column, without having to spell out the
entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
{quote}
Syntax:
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
add or replace
[columns (col1 int, col2 varchar)]
[properties ('prop1'='val1', 'prop2'='val2')]
{code}
Add command will add or replace existing columns or properties.
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
remove
[columns (`col1`, `col2`)]
[properties ('prop1', 'prop2')]
{code}
Remove command won't fail if column or property does not exist, will return
false status only.
was:
By [~Paul.Rogers]:
{quote}
Sooner or later users are going to ask for a command to update just the
properties, or just add or remove a column, without having to spell out the
entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
{quote}
Syntax:
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
add
[columns (col1 int, col2 varchar)]
[properties ('prop1'='val1', 'prop2'='val2')]
{code}
Add command will add or replace existing columns or properties.
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
remove
[columns (`col1`, `col2`)]
[properties ('prop1', 'prop2')]
{code}
Remove command won't fail if column or property does not exist, will return
false status only.
> Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
> --
>
> Key: DRILL-7168
> URL: https://issues.apache.org/jira/browse/DRILL-7168
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> By [~Paul.Rogers]:
> {quote}
> Sooner or later users are going to ask for a command to update just the
> properties, or just add or remove a column, without having to spell out the
> entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
> {quote}
> Syntax:
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> add or replace
> [columns (col1 int, col2 varchar)]
> [properties ('prop1'='val1', 'prop2'='val2')]
> {code}
> Add command will add or replace existing columns or properties.
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> remove
> [columns (`col1`, `col2`)]
> [properties ('prop1', 'prop2')]
> {code}
> Remove command won't fail if column or property does not exist, will return
> false status only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7168) Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
[
https://issues.apache.org/jira/browse/DRILL-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7168:
Description:
By [~Paul.Rogers]:
{quote}
Sooner or later users are going to ask for a command to update just the
properties, or just add or remove a column, without having to spell out the
entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
{quote}
Syntax:
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
add or replace
[columns (col1 int, col2 varchar)]
[properties ('prop1'='val1', 'prop2'='val2')]
{code}
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
remove
[columns (`col1`, `col2`)]
[properties ('prop1', 'prop2')]
{code}
Remove command won't fail if column or property does not exist, will return
false status only.
was:
By [~Paul.Rogers]:
{quote}
Sooner or later users are going to ask for a command to update just the
properties, or just add or remove a column, without having to spell out the
entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
{quote}
Syntax:
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
add or replace
[columns (col1 int, col2 varchar)]
[properties ('prop1'='val1', 'prop2'='val2')]
{code}
Add command will add or replace existing columns or properties.
{code:sql}
alter schema
(for table dfs.tmp.nation | path '/tmp/schema.json')
remove
[columns (`col1`, `col2`)]
[properties ('prop1', 'prop2')]
{code}
Remove command won't fail if column or property does not exist, will return
false status only.
> Implement ALTER SCHEMA ADD / REMOVE COLUMN / PROPERTY commands
> --
>
> Key: DRILL-7168
> URL: https://issues.apache.org/jira/browse/DRILL-7168
> Project: Apache Drill
> Issue Type: Sub-task
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> By [~Paul.Rogers]:
> {quote}
> Sooner or later users are going to ask for a command to update just the
> properties, or just add or remove a column, without having to spell out the
> entire new schema. ALTER TABLE SCHEMA ADD/REMOVE COLUMN/PROPERTY ...
> {quote}
> Syntax:
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> add or replace
> [columns (col1 int, col2 varchar)]
> [properties ('prop1'='val1', 'prop2'='val2')]
> {code}
> {code:sql}
> alter schema
> (for table dfs.tmp.nation | path '/tmp/schema.json')
> remove
> [columns (`col1`, `col2`)]
> [properties ('prop1', 'prop2')]
> {code}
> Remove command won't fail if column or property does not exist, will return
> false status only.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7273) Create operator for handling metadata
[ https://issues.apache.org/jira/browse/DRILL-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Volodymyr Vysotskyi updated DRILL-7273: --- Description: As described in the design document for [File metastore table metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] it is required to create an operator for handling metadata to be able to do incremental analyze. Add options: metastore.enabled metastore.metadata.store.depth_level Statement: ANALYZE TABLE REFRESH METADATA ... was: As described in the design document for [File metastore table metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] it is required to create an operator for handling metadata to be able to do incremental analyze. Add options: metastore.enabled metadata.store.depth_level Statement: ANALYZE TABLE REFRESH METADATA ... > Create operator for handling metadata > - > > Key: DRILL-7273 > URL: https://issues.apache.org/jira/browse/DRILL-7273 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Arina Ielchiieva >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: doc-impacting > Fix For: 1.17.0 > > > As described in the design document for [File metastore table > metadata|https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit#heading=h.j079wdhtehuk] > it is required to create an operator for handling metadata to be able to do > incremental analyze. > Add options: > metastore.enabled > metastore.metadata.store.depth_level > Statement: > ANALYZE TABLE REFRESH METADATA ... -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913450#comment-16913450
]
ASF GitHub Bot commented on DRILL-7356:
---
vvysotskyi commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846
For details please see
[DRILL-7356](https://issues.apache.org/jira/browse/DRILL-7356).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913457#comment-16913457
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316767216
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
Review comment:
```suggestion
* which will be stored in the Drill metastore. For example, when {@code
FILE} level value
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913463#comment-16913463
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316770040
##
File path: exec/java-exec/src/main/resources/drill-module.conf
##
@@ -691,5 +691,13 @@ drill.exec.options: {
# = rm related options ===
exec.rm.queryTags: "",
exec.rm.queues.wait_for_preferred_nodes: true,
-exec.statistics.tdigest_compression: 100
+exec.statistics.tdigest_compression: 100,
+# = metastore related options ===
Review comment:
```suggestion
# = Metastore related options ===
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913459#comment-16913459
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316767345
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
Review comment:
```suggestion
* is set, {@code ROW_GROUP} level metadata eon't be collected and stored
into the Metastore.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913456#comment-16913456
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316766564
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
Review comment:
```suggestion
* Option that enables Drill Metastore usage.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913460#comment-16913460
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316766755
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
Review comment:
```suggestion
* which will be stored in the Drill Metastore. For example, when {@code
FILE} value
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913458#comment-16913458
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316766693
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
Review comment:
```suggestion
new OptionDescription("Enables Drill Metastore usage to be able to
store table metadata " +
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913461#comment-16913461
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316768605
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
+ */
+ public static final String METASTORE_METADATA_STORE_DEPTH_LEVEL =
"metastore.metadata.store.depth_level";
+ public static final EnumeratedStringValidator
METASTORE_METADATA_STORE_DEPTH_LEVEL_VALIDATOR = new
EnumeratedStringValidator(METASTORE_METADATA_STORE_DEPTH_LEVEL,
+ new OptionDescription("Specifies maximum level depth for collecting
metadata. " +
+ "This option is not active for now. Default is 'ROW_GROUP'. (Drill
1.17+)"),
+ "TABLE", "SEGMENT", "PARTITION", "FILE", "ROW_GROUP");
+
+ /**
+ * Option for enabling usage of the schema stored to the metastore.
+ */
+ public static final String METASTORE_USE_SCHEMA_METADATA =
"metastore.metadata.use_schema";
+ public static final BooleanValidator METASTORE_USE_SCHEMA_METADATA_VALIDATOR
= new BooleanValidator(METASTORE_USE_SCHEMA_METADATA,
+ new OptionDescription("Enables usage of the schema stored to the
metastore. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for enabling usage of the statistics stored to the metastore at
the planning stage.
Review comment:
```suggestion
* Option for enabling statistics usage, stored in the Metastore, at the
planning stage.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913465#comment-16913465
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316769136
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
+ */
+ public static final String METASTORE_METADATA_STORE_DEPTH_LEVEL =
"metastore.metadata.store.depth_level";
+ public static final EnumeratedStringValidator
METASTORE_METADATA_STORE_DEPTH_LEVEL_VALIDATOR = new
EnumeratedStringValidator(METASTORE_METADATA_STORE_DEPTH_LEVEL,
+ new OptionDescription("Specifies maximum level depth for collecting
metadata. " +
+ "This option is not active for now. Default is 'ROW_GROUP'. (Drill
1.17+)"),
+ "TABLE", "SEGMENT", "PARTITION", "FILE", "ROW_GROUP");
+
+ /**
+ * Option for enabling usage of the schema stored to the metastore.
+ */
+ public static final String METASTORE_USE_SCHEMA_METADATA =
"metastore.metadata.use_schema";
+ public static final BooleanValidator METASTORE_USE_SCHEMA_METADATA_VALIDATOR
= new BooleanValidator(METASTORE_USE_SCHEMA_METADATA,
+ new OptionDescription("Enables usage of the schema stored to the
metastore. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for enabling usage of the statistics stored to the metastore at
the planning stage.
+ */
+ public static final String METASTORE_USE_STATISTICS_METADATA =
"metastore.metadata.use_statistics";
+ public static final BooleanValidator
METASTORE_USE_STATISTICS_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_USE_STATISTICS_METADATA,
+ new OptionDescription("Enables usage of the statistics stored to the
metastore at the planning stage. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for collecting schema and / or column statistics for every table
after CTAS and CTTAS execution.
+ */
+ public static final String METASTORE_CTAS_AUTO_COLLECT_METADATA =
"metastore.metadata.ctas.auto-collect";
+ public static final EnumeratedStringValidator
METASTORE_CTAS_AUTO_COLLECT_METADATA_VALIDATOR = new
EnumeratedStringValidator(METASTORE_CTAS_AUTO_COLLECT_METADATA,
+ new OptionDescription("Specifies whether schema and / or column
statistics will be " +
+ "automatically collected for every table after CTAS and CTTAS. " +
+ "This option is not active for now. Default is 'NONE'. (Drill
1.17+)"),
+ "NONE", "ALL", "SCHEMA");
+
+ /**
+ * Option for allowing using file metadata cache if required metadata is
absent in the metastore.
Review comment:
```suggestion
* Option for allowing using file metadata cache if required metadata is
absent in the Metastore.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volody
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913464#comment-16913464
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316768669
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
+ */
+ public static final String METASTORE_METADATA_STORE_DEPTH_LEVEL =
"metastore.metadata.store.depth_level";
+ public static final EnumeratedStringValidator
METASTORE_METADATA_STORE_DEPTH_LEVEL_VALIDATOR = new
EnumeratedStringValidator(METASTORE_METADATA_STORE_DEPTH_LEVEL,
+ new OptionDescription("Specifies maximum level depth for collecting
metadata. " +
+ "This option is not active for now. Default is 'ROW_GROUP'. (Drill
1.17+)"),
+ "TABLE", "SEGMENT", "PARTITION", "FILE", "ROW_GROUP");
+
+ /**
+ * Option for enabling usage of the schema stored to the metastore.
+ */
+ public static final String METASTORE_USE_SCHEMA_METADATA =
"metastore.metadata.use_schema";
+ public static final BooleanValidator METASTORE_USE_SCHEMA_METADATA_VALIDATOR
= new BooleanValidator(METASTORE_USE_SCHEMA_METADATA,
+ new OptionDescription("Enables usage of the schema stored to the
metastore. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for enabling usage of the statistics stored to the metastore at
the planning stage.
+ */
+ public static final String METASTORE_USE_STATISTICS_METADATA =
"metastore.metadata.use_statistics";
+ public static final BooleanValidator
METASTORE_USE_STATISTICS_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_USE_STATISTICS_METADATA,
+ new OptionDescription("Enables usage of the statistics stored to the
metastore at the planning stage. " +
Review comment:
Same here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913466#comment-16913466
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316769474
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
+ */
+ public static final String METASTORE_METADATA_STORE_DEPTH_LEVEL =
"metastore.metadata.store.depth_level";
+ public static final EnumeratedStringValidator
METASTORE_METADATA_STORE_DEPTH_LEVEL_VALIDATOR = new
EnumeratedStringValidator(METASTORE_METADATA_STORE_DEPTH_LEVEL,
+ new OptionDescription("Specifies maximum level depth for collecting
metadata. " +
+ "This option is not active for now. Default is 'ROW_GROUP'. (Drill
1.17+)"),
+ "TABLE", "SEGMENT", "PARTITION", "FILE", "ROW_GROUP");
+
+ /**
+ * Option for enabling usage of the schema stored to the metastore.
+ */
+ public static final String METASTORE_USE_SCHEMA_METADATA =
"metastore.metadata.use_schema";
+ public static final BooleanValidator METASTORE_USE_SCHEMA_METADATA_VALIDATOR
= new BooleanValidator(METASTORE_USE_SCHEMA_METADATA,
+ new OptionDescription("Enables usage of the schema stored to the
metastore. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for enabling usage of the statistics stored to the metastore at
the planning stage.
+ */
+ public static final String METASTORE_USE_STATISTICS_METADATA =
"metastore.metadata.use_statistics";
+ public static final BooleanValidator
METASTORE_USE_STATISTICS_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_USE_STATISTICS_METADATA,
+ new OptionDescription("Enables usage of the statistics stored to the
metastore at the planning stage. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for collecting schema and / or column statistics for every table
after CTAS and CTTAS execution.
+ */
+ public static final String METASTORE_CTAS_AUTO_COLLECT_METADATA =
"metastore.metadata.ctas.auto-collect";
+ public static final EnumeratedStringValidator
METASTORE_CTAS_AUTO_COLLECT_METADATA_VALIDATOR = new
EnumeratedStringValidator(METASTORE_CTAS_AUTO_COLLECT_METADATA,
+ new OptionDescription("Specifies whether schema and / or column
statistics will be " +
+ "automatically collected for every table after CTAS and CTTAS. " +
+ "This option is not active for now. Default is 'NONE'. (Drill
1.17+)"),
+ "NONE", "ALL", "SCHEMA");
+
+ /**
+ * Option for allowing using file metadata cache if required metadata is
absent in the metastore.
+ */
+ public static final String METASTORE_FALLBACK_TO_FILE_METADATA =
"metastore.metadata.fallback_to_file_metadata";
+ public static final BooleanValidator
METASTORE_FALLBACK_TO_FILE_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_FALLBACK_TO_FILE_METADATA,
+ new OptionDescription("Allows using file metadata cache for the case
when required metadata is absent in the metastore. " +
+ "This option is not active for now. Default is true. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying the number of attempts for retrying query planning
after detecting that query metadata is changed.
+ */
+ public static final String METASTORE_RETRIVAL_RETRY_ATTEMPTS =
"metastore.retrival.retry_attempts";
+ public static final IntegerValidator
METASTORE_RETRIVAL_RETRY_ATTEMPT
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913462#comment-16913462
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316769216
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
+ */
+ public static final String METASTORE_METADATA_STORE_DEPTH_LEVEL =
"metastore.metadata.store.depth_level";
+ public static final EnumeratedStringValidator
METASTORE_METADATA_STORE_DEPTH_LEVEL_VALIDATOR = new
EnumeratedStringValidator(METASTORE_METADATA_STORE_DEPTH_LEVEL,
+ new OptionDescription("Specifies maximum level depth for collecting
metadata. " +
+ "This option is not active for now. Default is 'ROW_GROUP'. (Drill
1.17+)"),
+ "TABLE", "SEGMENT", "PARTITION", "FILE", "ROW_GROUP");
+
+ /**
+ * Option for enabling usage of the schema stored to the metastore.
+ */
+ public static final String METASTORE_USE_SCHEMA_METADATA =
"metastore.metadata.use_schema";
+ public static final BooleanValidator METASTORE_USE_SCHEMA_METADATA_VALIDATOR
= new BooleanValidator(METASTORE_USE_SCHEMA_METADATA,
+ new OptionDescription("Enables usage of the schema stored to the
metastore. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for enabling usage of the statistics stored to the metastore at
the planning stage.
+ */
+ public static final String METASTORE_USE_STATISTICS_METADATA =
"metastore.metadata.use_statistics";
+ public static final BooleanValidator
METASTORE_USE_STATISTICS_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_USE_STATISTICS_METADATA,
+ new OptionDescription("Enables usage of the statistics stored to the
metastore at the planning stage. " +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for collecting schema and / or column statistics for every table
after CTAS and CTTAS execution.
+ */
+ public static final String METASTORE_CTAS_AUTO_COLLECT_METADATA =
"metastore.metadata.ctas.auto-collect";
+ public static final EnumeratedStringValidator
METASTORE_CTAS_AUTO_COLLECT_METADATA_VALIDATOR = new
EnumeratedStringValidator(METASTORE_CTAS_AUTO_COLLECT_METADATA,
+ new OptionDescription("Specifies whether schema and / or column
statistics will be " +
+ "automatically collected for every table after CTAS and CTTAS. " +
+ "This option is not active for now. Default is 'NONE'. (Drill
1.17+)"),
+ "NONE", "ALL", "SCHEMA");
+
+ /**
+ * Option for allowing using file metadata cache if required metadata is
absent in the metastore.
+ */
+ public static final String METASTORE_FALLBACK_TO_FILE_METADATA =
"metastore.metadata.fallback_to_file_metadata";
+ public static final BooleanValidator
METASTORE_FALLBACK_TO_FILE_METADATA_VALIDATOR = new
BooleanValidator(METASTORE_FALLBACK_TO_FILE_METADATA,
+ new OptionDescription("Allows using file metadata cache for the case
when required metadata is absent in the metastore. " +
Review comment:
```suggestion
new OptionDescription("Allows using file metadata cache for the case
when required metadata is absent in the Metastore. " +
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
[jira] [Created] (DRILL-7358) Text reader returns nothing for count queries over empty files
Arina Ielchiieva created DRILL-7358:
---
Summary: Text reader returns nothing for count queries over empty
files
Key: DRILL-7358
URL: https://issues.apache.org/jira/browse/DRILL-7358
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.16.0
Reporter: Arina Ielchiieva
Assignee: Paul Rogers
Fix For: 1.17.0
If we do count over empty CSV files (with or without headers), there is not
result.
Though expecting result is 0.
Unit tests examples:
{code}
@Test
public void testCount() throws Exception {
String fileName = "headersOnly.csv";
try (PrintWriter out = new PrintWriter(new FileWriter(new File(testDir,
fileName {
out.print("a,b,c"); // note: no \n in the end
}
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
{code}
@Test
public void testCount() throws Exception {
String fileName = "empty.csv";
File file = new File(testDir, fileName);
assertTrue(file.createNewFile());
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913474#comment-16913474
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on pull request #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#discussion_r316767345
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java
##
@@ -1044,4 +1044,67 @@ public static String bootDefaultFor(String name) {
public static final RangeLongValidator QUERY_MAX_ROWS_VALIDATOR = new
RangeLongValidator(QUERY_MAX_ROWS, 0, Integer.MAX_VALUE,
new OptionDescription("The maximum number of rows that the query will
return. This can be only set at a SYSTEM level by an admin. (Drill 1.16+)"));
+ /**
+ * Option that enables Drill metastore usage.
+ */
+ public static final String METASTORE_ENABLED = "metastore.enabled";
+ public static final BooleanValidator METASTORE_ENABLED_VALIDATOR = new
BooleanValidator(METASTORE_ENABLED,
+ new OptionDescription("Enables Drill metastore usage to be able to store
table metadata " +
+ "during ANALYZE TABLE commands execution and to be able to read
table metadata during regular " +
+ "queries execution or when querying some INFORMATION_SCHEMA tables.
" +
+ "This option is not active for now. Default is false. (Drill
1.17+)"));
+
+ /**
+ * Option for specifying maximum level depth for collecting metadata
+ * which will be stored in the Drill metastore. For example, when {@code
FILE} value
+ * is set, {@code ROW_GROUP} level metadata wouldn't be collected and stored
into the metastore.
Review comment:
```suggestion
* is set, {@code ROW_GROUP} level metadata won't be collected and stored
into the Metastore.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913480#comment-16913480
]
ASF GitHub Bot commented on DRILL-7356:
---
vvysotskyi commented on issue #1846: DRILL-7356: Introduce session options for
the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#issuecomment-523979827
@arina-ielchiieva, thanks for the review, I have addressed CR comments.
Please take a look again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913484#comment-16913484
]
ASF GitHub Bot commented on DRILL-7356:
---
arina-ielchiieva commented on issue #1846: DRILL-7356: Introduce session
options for the Drill Metastore
URL: https://github.com/apache/drill/pull/1846#issuecomment-523982382
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7356) Introduce session options for the Drill Metastore
[
https://issues.apache.org/jira/browse/DRILL-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7356:
Labels: ready-to-commit (was: )
> Introduce session options for the Drill Metastore
> -
>
> Key: DRILL-7356
> URL: https://issues.apache.org/jira/browse/DRILL-7356
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.17.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> Introduce a new session option for enabling metastore: {{metastore.enabled}}.
> It will be used during the next steps of development.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (DRILL-7358) Text reader returns nothing for count queries over empty files
[
https://issues.apache.org/jira/browse/DRILL-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7358:
Description:
If we do count over empty CSV files (with or without headers), there is no
result.
Though expecting result is 0.
Unit tests examples:
{code}
@Test
public void testCount() throws Exception {
String fileName = "headersOnly.csv";
try (PrintWriter out = new PrintWriter(new FileWriter(new File(testDir,
fileName {
out.print("a,b,c"); // note: no \n in the end
}
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
{code}
@Test
public void testCount() throws Exception {
String fileName = "empty.csv";
File file = new File(testDir, fileName);
assertTrue(file.createNewFile());
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
was:
If we do count over empty CSV files (with or without headers), there is not
result.
Though expecting result is 0.
Unit tests examples:
{code}
@Test
public void testCount() throws Exception {
String fileName = "headersOnly.csv";
try (PrintWriter out = new PrintWriter(new FileWriter(new File(testDir,
fileName {
out.print("a,b,c"); // note: no \n in the end
}
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
{code}
@Test
public void testCount() throws Exception {
String fileName = "empty.csv";
File file = new File(testDir, fileName);
assertTrue(file.createNewFile());
queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
"`").print();
}
{code}
> Text reader returns nothing for count queries over empty files
> --
>
> Key: DRILL-7358
> URL: https://issues.apache.org/jira/browse/DRILL-7358
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Arina Ielchiieva
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.17.0
>
>
> If we do count over empty CSV files (with or without headers), there is no
> result.
> Though expecting result is 0.
> Unit tests examples:
> {code}
> @Test
> public void testCount() throws Exception {
> String fileName = "headersOnly.csv";
> try (PrintWriter out = new PrintWriter(new FileWriter(new File(testDir,
> fileName {
> out.print("a,b,c"); // note: no \n in the end
> }
> queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
> "`").print();
> }
> {code}
> {code}
> @Test
> public void testCount() throws Exception {
> String fileName = "empty.csv";
> File file = new File(testDir, fileName);
> assertTrue(file.createNewFile());
> queryBuilder().sql("SELECT count(1) FROM `dfs.data`.`" + fileName +
> "`").print();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913506#comment-16913506
]
ASF GitHub Bot commented on DRILL-7222:
---
kkhatua commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r316787482
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -587,6 +622,25 @@
if (e.target.form)
<#if
model.isOnlyImpersonationEnabled()>doSubmitQueryWithUserName()<#else>doSubmitQueryWithAutoLimit();
});
+
+// Extract estimated rowcount map
+var opRowCountMap = {};
+// Get OpId-Rowocunt Map
+function buildRowCountMap() {
+ var phyText = $('#query-physical').find('pre').text();
+ var opLines = phyText.split("\n");
+ opLines.forEach(line => {
+if (line.trim().length > 0) {
+ var opId = line.match(/\d+-\d+/g)[0];
+ var opRowCount = line.match(/rowcount = ([^,]+)/)[1];
+ if (opRowCount.includes("E")) {
Review comment:
Sorry... missed this in the email notification. Done now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913720#comment-16913720 ] ASF GitHub Bot commented on DRILL-7222: --- agozhiy commented on issue #1779: DRILL-7222: Visualize estimated and actual row counts for a query URL: https://github.com/apache/drill/pull/1779#issuecomment-524092943 +1. Thanks for the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Gozhiy updated DRILL-7222: Labels: doc-impacting ready-to-commit user-experience (was: doc-impacting user-experience) > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, ready-to-commit, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
