[
https://issues.apache.org/jira/browse/HCATALOG-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202791#comment-13202791
]
[email protected] commented on HCATALOG-240:
--------------------------------------------------------
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hadoop/mapred/HCatMapRedUtil.java, line 1
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72848#file72848line1>
bq. >
bq. > It seems odd to put this into the o.a.hadoop.mapred package. Are
you calling some package scoped function or something?
Sadly yes. mapred.JobContext and mapred.TaskAttemptContext both have
package-private constructor. Nasty but ound no other way around it.
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hcatalog/common/HCatUtil.java, line 465
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72850#file72850line465>
bq. >
bq. > Some javadoc here on what should be passed into each of these values
would be helpful. They are class names I assume?
Yep. Will add javadoc in the next patch. As I've mentioned in the jira there
are a few things that are missing. I'll make sure this gets documented.
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hcatalog/common/HCatUtil.java, line 509
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72850#file72850line509>
bq. >
bq. > We shouldn't need code at all to pull out ISD and OSD info, since
we're removing them, correct? So I'm confused why a new function is being
added here to pull that info out.
Yeah I'll remove it. OutputStorageDriver cleansing is only partially done since
so a lot of things will break. Probably good to do that once everything is
committed and working.
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hcatalog/mapreduce/FileRecordWriterContainer.java,
line 161
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72857#file72857line161>
bq. >
bq. > What is better that needs to be put here?
I'll be more elaborate. We shouldn't pass a NULL reporter as much as possible
or else the underlying recorderWriter won't be able to report progress to the
task tracker. One of the things I still have to do.
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hcatalog/mapreduce/FileRecordWriterContainer.java,
line 264
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72857#file72857line264>
bq. >
bq. > We should add this before we commit the patch, as it should speed
things up quite a bit.
I was hoping to make that change during integration with Sushanth's HCatRecord
patch. Should be easy enough to do before that as well.
bq. On 2012-02-07 19:01:36, Alan Gates wrote:
bq. > src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java, line 27
bq. > <https://reviews.apache.org/r/3775/diff/1/?file=72858#file72858line27>
bq. >
bq. > Some javadoc on the purpose of this class would be good. I'm
assuming the purpose is to be a stand in StorageHandler in the HDFS storage
case where there isn't a storage handler.
bq. >
bq. > Once we have moved getAuthorizationProvider from HCatStorageHandler
to HiveStorageHandler will this class extend HiveStorageHandler?
Correct, it's a StorageHandler for orphan serDe,IF,OF groups.
There are four methods the HCatStorageHandler add
1. configureOuputJobProperties()
2. configureInputJobProperties()
3. getAuthorizationProvider()
4. getOuputFormatContainer()
The first three methods HCatStorageHandler is only standing in for
HiveStorageHandler. The 4th is used to determine what type of storage system
the storageHandler represents. Hive doesn't have this and that's why they had
to do a lot of hacky things to get HBase working (ie have HBaseSplit extend
FileSplit) even though they have some notion of containers (ie
HiveOutputFormat). We can have this as a stand in for now until we decide on
the final solution HiveStorageHandler will implement.
- Francis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3775/#review4867
-----------------------------------------------------------
On 2012-02-07 17:27:58, Francis Liu wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3775/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-07 17:27:58)
bq.
bq.
bq. Review request for hcatalog, Alan Gates and Sushanth Sowmyan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. First drop. See HCATALOG-240.
bq.
bq.
bq. This addresses bug HCATALOG-240.
bq. https://issues.apache.org/jira/browse/HCATALOG-240
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java
f37c5fd
bq. src/java/org/apache/hcatalog/mapreduce/DefaultOutputFormatContainer.java
13fa8ac
bq. src/java/org/apache/hcatalog/mapreduce/DefaultRecordWriterContainer.java
7c90737
bq.
src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java
9c5fdd3
bq. src/java/org/apache/hadoop/mapred/HCatMapRedUtil.java PRE-CREATION
bq. src/java/org/apache/hcatalog/cli/SemanticAnalysis/CreateTableHook.java
9d98f50
bq. src/java/org/apache/hcatalog/common/HCatUtil.java 13b56fd
bq. src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java
e91ed0f
bq. src/java/org/apache/hcatalog/mapreduce/FileOutputStorageDriver.java
6596264
bq. src/java/org/apache/hcatalog/mapreduce/FileRecordWriterContainer.java
c72cb4f
bq. src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/HCatBaseOutputFormat.java 1b2f9a4
bq. src/java/org/apache/hcatalog/mapreduce/HCatEximInputFormat.java 90c5671
bq. src/java/org/apache/hcatalog/mapreduce/HCatEximInputFormat.java.broken
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/HCatEximOutputCommitter.java
1ba8ccd
bq.
src/java/org/apache/hcatalog/mapreduce/HCatEximOutputCommitter.java.broken
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/HCatEximOutputFormat.java ed1ba66
bq. src/java/org/apache/hcatalog/mapreduce/HCatEximOutputFormat.java.broken
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java d7eab9e
bq. src/java/org/apache/hcatalog/mapreduce/HCatOutputStorageDriver.java
566943f
bq. src/java/org/apache/hcatalog/mapreduce/HCatStorageHandler.java
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/HCatTableInfo.java e65f1d0
bq. src/java/org/apache/hcatalog/mapreduce/OutputCommitterContainer.java
5f0585a
bq. src/java/org/apache/hcatalog/mapreduce/OutputFormatContainer.java
f73a6dc
bq. src/java/org/apache/hcatalog/mapreduce/OutputJobInfo.java 9830190
bq. src/java/org/apache/hcatalog/mapreduce/ProgressReporter.java
PRE-CREATION
bq. src/java/org/apache/hcatalog/mapreduce/RecordWriterContainer.java
9a93bfa
bq. src/java/org/apache/hcatalog/mapreduce/StorerInfo.java 8785ffb
bq. src/java/org/apache/hcatalog/pig/HCatEximLoader.java a36f808
bq. src/java/org/apache/hcatalog/pig/HCatEximLoader.java.broken PRE-CREATION
bq. src/java/org/apache/hcatalog/pig/HCatEximStorer.java b7bee44
bq. src/java/org/apache/hcatalog/pig/HCatEximStorer.java.broken PRE-CREATION
bq. src/test/org/apache/hcatalog/cli/DummyStorageHandler.java 0e9565c
bq. src/test/org/apache/hcatalog/cli/TestStorageHandlerProperties.java
df8f5a5
bq.
src/test/org/apache/hcatalog/cli/TestStorageHandlerProperties.java.broken
PRE-CREATION
bq. src/test/org/apache/hcatalog/mapreduce/TestHCatEximInputFormat.java
9d94c7b
bq.
src/test/org/apache/hcatalog/mapreduce/TestHCatEximInputFormat.java.broken
PRE-CREATION
bq. src/test/org/apache/hcatalog/mapreduce/TestHCatEximOutputFormat.java
99ec02b
bq.
src/test/org/apache/hcatalog/mapreduce/TestHCatEximOutputFormat.java.broken
PRE-CREATION
bq. src/test/org/apache/hcatalog/mapreduce/TestHCatOutputFormat.java 5df0146
bq. src/test/org/apache/hcatalog/pig/TestHCatEximLoader.java 7691fb9
bq. src/test/org/apache/hcatalog/pig/TestHCatEximLoader.java.broken
PRE-CREATION
bq. src/test/org/apache/hcatalog/pig/TestHCatEximStorer.java c3ba19d
bq. src/test/org/apache/hcatalog/pig/TestHCatEximStorer.java.broken
PRE-CREATION
bq. storage-drivers/build.xml efd26d0
bq.
bq. Diff: https://reviews.apache.org/r/3775/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Francis
bq.
bq.
> Changes to HCatOutputFormat to make it use SerDes instead of StorageDriver
> --------------------------------------------------------------------------
>
> Key: HCATALOG-240
> URL: https://issues.apache.org/jira/browse/HCATALOG-240
> Project: HCatalog
> Issue Type: Sub-task
> Components: storage handlers
> Affects Versions: 0.4
> Reporter: Alan Gates
> Assignee: Francis Liu
> Fix For: 0.4
>
> Attachments: HCATALOG-240.patch
>
>
> This JIRA covers changes to HCatOutputFormat, OutputJobInfo, and
> HCatOutputCommitter. See HCATALOG-237 for design notes and details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
