[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283889#comment-16283889 ] Sergey Shelukhin commented on HIVE-11553: - Actually metadata cache work is on hold because HBase metastore got abandoned, so it should be ok to remove the dependency if needed. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219572#comment-16219572 ] Naveen Gangam commented on HIVE-11553: -- Hey [~sershe], quick follow up on this. Was there a reason we are packaging thrift classes *fb303* into hive-exec.jar as part of this change? In the older releases, we do not include these classes. Thanks > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219664#comment-16219664 ] Sergey Shelukhin commented on HIVE-11553: - I think the reason is that it is a metastore dependency. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734229#comment-15734229 ] Mithun Radhakrishnan commented on HIVE-11553: - This patch seems to have renamed {{HiveConf.ConfVars.METASTORE_BATCH_RETRIEVE_TABLE_PARTITION_MAX}} to {{HiveConf.ConfVars.METASTORE_BATCH_RETRIEVE_OBJECTS_MAX}}, instead of deprecating it in favour of a new constant. :/ > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901146#comment-14901146 ] Prasanth Jayachandran commented on HIVE-11553: -- Some comments: 1) Do we need another config for batch size? hive.metastore.batch.retrieve.file.metadata.max. Can we just have only one? something like batch.retrieve.size? Also we don't have min threshold I believe. 2) Fix description for HIVE_ORC_MS_FOOTER_CACHE_ENABLED config 3) Can you please javadoc for the newly added interfaces in IMetaStoreClient? 4) Can you explain what's going on with read-only bytebuffers? Still looking into OrcInputFormat changes > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901328#comment-14901328 ] Prasanth Jayachandran commented on HIVE-11553: -- More comments: 1) // TODO: is this really needed? Shouldn't this be set already?.. I don't think so. Only fileMetaInfo can change based on this config HIVE_ORC_INCLUDE_FILE_FOOTER_IN_SPLITS. metadata,types and writerVersion shouldn't change. Can you remove it? 2) What is happening in catch block for createFileInfoFromMs()? If the bytebuffer is corrupted for some reason why not just throw it away,invalidate cache and read it from file (and cache it again)? 3) Why do we need to duplicate the buffer in createFileInfoFromMs()? 4) Also remove the temporary method makeLongKey() if point 2) is addressed? > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901334#comment-14901334 ] Prasanth Jayachandran commented on HIVE-11553: -- Also, can you create followup jiras for the TODOs like batching the calls, removing unwanted code etc. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905581#comment-14905581 ] Sergey Shelukhin commented on HIVE-11553: - 1.4) See THRIFT-3352 > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905601#comment-14905601 ] Sergey Shelukhin commented on HIVE-11553: - 2.3) for logging in case of error. Note that it doesn't copy the buffer. The rest - done > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905916#comment-14905916 ] Prasanth Jayachandran commented on HIVE-11553: -- Few more comments in RB. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906346#comment-14906346 ] Hive QA commented on HIVE-11553: {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12762019/HIVE-11553.04.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5397/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5397/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5397/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-5397/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 648f2c6 HIVE-11922: Better error message when ORC split generation fails (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git clean -f -d Removing ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java.orig + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at 648f2c6 HIVE-11922: Better error message when ORC split generation fails (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12762019 - PreCommit-HIVE-TRUNK-Build > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938178#comment-14938178 ] Prasanth Jayachandran commented on HIVE-11553: -- [~sershe] Who is creating read-only ByteBuffer? Why not create a writeable BB and pass around to avoid reflect/copy? I don't see the hashmap being initialized with size in your latest patch. Otherwise, +1. Pending tests. You may have to rebase the patch. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938209#comment-14938209 ] Sergey Shelukhin commented on HIVE-11553: - I will remove reflection I guess and add a TODO. Buffer is coming from HBase client. Which hashtable? > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938215#comment-14938215 ] Prasanth Jayachandran commented on HIVE-11553: -- {code} HashMap posMap = new HashMap<>(); {code} could be initialized with files.size(). > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940735#comment-14940735 ] Hive QA commented on HIVE-11553: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12764477/HIVE-11553.06.patch {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9641 tests executed *Failed tests:* {noformat} org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5490/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5490/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5490/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12764477 - PreCommit-HIVE-TRUNK-Build > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.06.patch, HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942170#comment-14942170 ] Lefty Leverenz commented on HIVE-11553: --- Doc note: This adds configuration parameter *hive.orc.splits.ms.footer.cache.enabled* and changes the description of *hive.metastore.batch.retrieve.table.partition.max* which was introduced by HIVE-2907 in 0.10.0 but isn't documented yet. Here's where they should be documented: * [Configuration Properties -- ORC File Format | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-ORCFileFormat] * [Configuration Properties -- MetaStore | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore] After the status gets updated to show that this was committed to master, a TODOC2.0 label should be added. (The commit ID is bc1c434f1ffc70dc2d52f34839c6f54b0b0c8d10.) > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15043540#comment-15043540 ] Lefty Leverenz commented on HIVE-11553: --- See Sergey's comment on HIVE-12602 for more information about *hive.orc.splits.ms.footer.cache.enabled*. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.03.patch, HIVE-11553.04.patch, HIVE-11553.06.patch, > HIVE-11553.07.patch, HIVE-11553.patch > > > This is the first step; uses the simple footer-getting API, without PPD. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711762#comment-14711762 ] Sergey Shelukhin commented on HIVE-11553: - The dependencies for metastore client need to be propagated to Tez AM, also there are some mysterious config propagation errors Other than that no changes to the patch so far. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715920#comment-14715920 ] Sergey Shelukhin commented on HIVE-11553: - There are a bunch of offset errors in this patch and in ORC based on current assumptions. I fixed some of them, will fix others later because my head hurts for today. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715931#comment-14715931 ] Sergey Shelukhin commented on HIVE-11553: - So far putting works, but getting sometimes fails because of wrong offsets and stuff > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720813#comment-14720813 ] Sergey Shelukhin commented on HIVE-11553: - Actually, I misread some code. Something is easier to do than I thought. I may yet update this patch. It does work now, still :) > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11553) use basic file metadata cache in ETLSplitStrategy-related paths
[ https://issues.apache.org/jira/browse/HIVE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791429#comment-14791429 ] Sergey Shelukhin commented on HIVE-11553: - [~gopalv] [~prasanth_j] can you please review this? Note that this is stage 1, before PPD. PPD is stage 2 :) Unfortunately my local branches are a clusterfuck by now and everything now depends on this patch, so makes it hard to make progress. > use basic file metadata cache in ETLSplitStrategy-related paths > --- > > Key: HIVE-11553 > URL: https://issues.apache.org/jira/browse/HIVE-11553 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-metastore-branch > > Attachments: HIVE-11553.01.patch, HIVE-11553.02.patch, > HIVE-11553.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)