[
https://issues.apache.org/jira/browse/FALCON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant updated FALCON-2234:
-----------------------------
Description:
Falcon feed is failing with org.apache.falcon.entity.parser.ValidationException
when extract type is incremental.
As per the Apache doc Falcon supports
Import policy
The valid combinations are:
[full,snapshot] - data is extracted in full and dumped into the feed instance
location.
[incremental, append] - data is extracted incrementally using the key specified
in the deltacolumn
As per the below code , it appears we only check [full,snapshot] and
[incremental, append] goes to exception
https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
Snippet
{code}
/**
* Validate extraction and merge type combination. Currently supported
combo:
*
* ExtractionType = FULL and MergeType = SNAPSHOT.
* ExtractionType = INCREMENTAL and MergeType = APPEND.
*
* @param feed Feed entity
* @param cluster Cluster referenced in the Feed definition
* @throws FalconException
*/
private void validateFeedExtractionType(Feed feed, Cluster cluster) throws
FalconException {
Extract extract = cluster.getImport().getSource().getExtract();
if (ExtractMethod.FULL == extract.getType()) {
if ((MergeType.SNAPSHOT != extract.getMergepolicy())
|| (extract.getDeltacolumn() != null)) {
throw new ValidationException(String.format("Feed %s is using
FULL "
+ "extract method but specifies either a superfluous "
+ "deltacolumn or a mergepolicy other than snapshot",
feed.getName()));
}
} else {
throw new ValidationException(String.format("Feed %s is using
unsupported "
+ "extraction mechanism %s", feed.getName(),
extract.getType().value()));
}
}
{code}
It must have a clause to also check [incremental, append] .
was:
Falcon feed is failing with org.apache.falcon.entity.parser.ValidationException
when extract type is incremental.
As per the Apache doc Falcon supports
Import policy
The valid combinations are:
[full,snapshot] - data is extracted in full and dumped into the feed instance
location.
[incremental, append] - data is extracted incrementally using the key specified
in the deltacolumn
As per the below code , it appears we only check [full,snapshot] and
[incremental, append] goes to exception
https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
Snippet
/*
private void validateFeedExtractionType(Feed feed, Cluster cluster) throws
FalconException {
Extract extract = cluster.getImport().getSource().getExtract();
if (ExtractMethod.FULL == extract.getType())
{
if ((MergeType.SNAPSHOT != extract.getMergepolicy())
|| (extract.getDeltacolumn() != null))
{
throw new ValidationException(String.format("Feed %s is using
FULL "
+ "extract method but specifies either a superfluous "
+ "deltacolumn or a mergepolicy other than snapshot",
feed.getName()));
}
}
else {
throw new ValidationException(String.format("Feed %s is using
unsupported "
+ "extraction mechanism %s", feed.getName(),
extract.getType().value()));
}
}
*/
It must have a clause to also check [incremental, append] .
> Falcon feed fails for RDBMS import if extract type is incremental
> -----------------------------------------------------------------
>
> Key: FALCON-2234
> URL: https://issues.apache.org/jira/browse/FALCON-2234
> Project: Falcon
> Issue Type: Bug
> Components: falcon-ui
> Affects Versions: 0.10
> Reporter: Prashant
> Priority: Critical
>
> Falcon feed is failing with
> org.apache.falcon.entity.parser.ValidationException
> when extract type is incremental.
> As per the Apache doc Falcon supports
> Import policy
> The valid combinations are:
> [full,snapshot] - data is extracted in full and dumped into the feed instance
> location.
> [incremental, append] - data is extracted incrementally using the key
> specified in the deltacolumn
> As per the below code , it appears we only check [full,snapshot] and
> [incremental, append] goes to exception
> https://github.com/apache/falcon/blob/0.10/common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java
> Snippet
> {code}
> /**
> * Validate extraction and merge type combination. Currently supported
> combo:
> *
> * ExtractionType = FULL and MergeType = SNAPSHOT.
> * ExtractionType = INCREMENTAL and MergeType = APPEND.
> *
> * @param feed Feed entity
> * @param cluster Cluster referenced in the Feed definition
> * @throws FalconException
> */
> private void validateFeedExtractionType(Feed feed, Cluster cluster)
> throws FalconException {
> Extract extract = cluster.getImport().getSource().getExtract();
> if (ExtractMethod.FULL == extract.getType()) {
> if ((MergeType.SNAPSHOT != extract.getMergepolicy())
> || (extract.getDeltacolumn() != null)) {
> throw new ValidationException(String.format("Feed %s is using
> FULL "
> + "extract method but specifies either a superfluous "
> + "deltacolumn or a mergepolicy other than
> snapshot", feed.getName()));
> }
> } else {
> throw new ValidationException(String.format("Feed %s is using
> unsupported "
> + "extraction mechanism %s", feed.getName(),
> extract.getType().value()));
> }
> }
> {code}
> It must have a clause to also check [incremental, append] .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
