[jira] [Updated] (HUDI-1174) Hudi changes for bootstrapped tables integration with Presto
[ https://issues.apache.org/jira/browse/HUDI-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha updated HUDI-1174: Status: Open (was: New) > Hudi changes for bootstrapped tables integration with Presto > > > Key: HUDI-1174 > URL: https://issues.apache.org/jira/browse/HUDI-1174 > Project: Apache Hudi > Issue Type: Sub-task > Components: bootstrap >Reporter: Udit Mehrotra >Assignee: Udit Mehrotra >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Hudi changes for bootstrapped tables integration with Presto. > * Annotation *UseRecordReaderFromInputFormat* is required on > *HoodieParquetInputFormat* as well, because the reading for bootstrapped > tables needs to happen through record reader to be able to perform the merge. > On presto side, this annotation is already handled. > * We need to internally maintain *VIRTUAL_COLUMN_NAMES* because presto's > internal hive version *hive-apache-1.2.2* has *VirutalColumn* as a *class*, > versus the one we depend on in hudi which is an *enum*. This results in > following error in presto: > > {noformat} > 2020-08-10T21:59:58.957Z ERROR remote-task-callback-2 > com.facebook.presto.execution.StageExecutionStateMachine Stage execution > 20200810_215953_6_34kqg.1.0 failed > java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.lambda$getRecordReader$2(HoodieParquetInputFormat.java:201) > at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) > at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:203) > at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:253) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:74) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.lambda$executeActionInDoAs$0(UserGroupInformationUtils.java:29) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1824) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.executeActionInDoAs(UserGroupInformationUtils.java:27) > at > com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39) > at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:82) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:73) > at > com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:374) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:137) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:113) > at > com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:52) > {noformat} > > * Dependency changes in *hudi-presto-bundle* to avoid runtime exceptions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1174) Hudi changes for bootstrapped tables integration with Presto
[ https://issues.apache.org/jira/browse/HUDI-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhavani Sudha updated HUDI-1174: Status: In Progress (was: Open) > Hudi changes for bootstrapped tables integration with Presto > > > Key: HUDI-1174 > URL: https://issues.apache.org/jira/browse/HUDI-1174 > Project: Apache Hudi > Issue Type: Sub-task > Components: bootstrap >Reporter: Udit Mehrotra >Assignee: Udit Mehrotra >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Hudi changes for bootstrapped tables integration with Presto. > * Annotation *UseRecordReaderFromInputFormat* is required on > *HoodieParquetInputFormat* as well, because the reading for bootstrapped > tables needs to happen through record reader to be able to perform the merge. > On presto side, this annotation is already handled. > * We need to internally maintain *VIRTUAL_COLUMN_NAMES* because presto's > internal hive version *hive-apache-1.2.2* has *VirutalColumn* as a *class*, > versus the one we depend on in hudi which is an *enum*. This results in > following error in presto: > > {noformat} > 2020-08-10T21:59:58.957Z ERROR remote-task-callback-2 > com.facebook.presto.execution.StageExecutionStateMachine Stage execution > 20200810_215953_6_34kqg.1.0 failed > java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.lambda$getRecordReader$2(HoodieParquetInputFormat.java:201) > at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) > at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:203) > at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:253) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:74) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.lambda$executeActionInDoAs$0(UserGroupInformationUtils.java:29) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1824) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.executeActionInDoAs(UserGroupInformationUtils.java:27) > at > com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39) > at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:82) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:73) > at > com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:374) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:137) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:113) > at > com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:52) > {noformat} > > * Dependency changes in *hudi-presto-bundle* to avoid runtime exceptions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1174) Hudi changes for bootstrapped tables integration with Presto
[ https://issues.apache.org/jira/browse/HUDI-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-1174: - Labels: pull-request-available (was: ) > Hudi changes for bootstrapped tables integration with Presto > > > Key: HUDI-1174 > URL: https://issues.apache.org/jira/browse/HUDI-1174 > Project: Apache Hudi > Issue Type: Sub-task > Components: bootstrap >Reporter: Udit Mehrotra >Assignee: Udit Mehrotra >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Hudi changes for bootstrapped tables integration with Presto. > * Annotation *UseRecordReaderFromInputFormat* is required on > *HoodieParquetInputFormat* as well, because the reading for bootstrapped > tables needs to happen through record reader to be able to perform the merge. > On presto side, this annotation is already handled. > * We need to internally maintain *VIRTUAL_COLUMN_NAMES* because presto's > internal hive version *hive-apache-1.2.2* has *VirutalColumn* as a *class*, > versus the one we depend on in hudi which is an *enum*. This results in > following error in presto: > > {noformat} > 2020-08-10T21:59:58.957Z ERROR remote-task-callback-2 > com.facebook.presto.execution.StageExecutionStateMachine Stage execution > 20200810_215953_6_34kqg.1.0 failed > java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.lambda$getRecordReader$2(HoodieParquetInputFormat.java:201) > at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) > at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) > at > org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:203) > at com.facebook.presto.hive.HiveUtil.createRecordReader(HiveUtil.java:253) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$0(GenericHiveRecordCursorProvider.java:74) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.lambda$executeActionInDoAs$0(UserGroupInformationUtils.java:29) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1824) > at > com.facebook.presto.hive.authentication.UserGroupInformationUtils.executeActionInDoAs(UserGroupInformationUtils.java:27) > at > com.facebook.presto.hive.authentication.ImpersonatingHdfsAuthentication.doAs(ImpersonatingHdfsAuthentication.java:39) > at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:82) > at > com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:73) > at > com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:374) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:137) > at > com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:113) > at > com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:52) > {noformat} > > * Dependency changes in *hudi-presto-bundle* to avoid runtime exceptions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)