[
https://issues.apache.org/jira/browse/GOBBLIN-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Sen updated GOBBLIN-1399:
-----------------------------
Description:
currently gobblin has hard coded specification for writer's path and file name
primarily it has namespace and tablename and default - 3 way to have writer's
path and file name.
{code:java}
// code placeholder
switch (getWriterFilePathType(state)) {
case NAMESPACE_TABLE:
// writer.file.path.format = <extract.namespace>/<extract.table.name>/
return getNamespaceTableWriterFilePath(state);
case TABLENAME:
// <extract.table.name>
return WriterUtils.getTableNameWriterFilePath(state);
default:
return WriterUtils.getDefaultWriterFilePath(state, numBranches, branchId);
}
{code}
Filename:
{code:java}
namespace.replaceAll("\\.", "/") + "/" + table + "/" + extractId + "_"
+ (isFull ? "full" : "append");
{code}
There is no way user can add any other parameters like version, batchId.
Also it would be awesome to have any configuration value to be part of the
writer path, which can be defined by the format like this
{code}
Unable to find source-code formatter for language: java. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
yamlextract.type = increments
writer.file.path.format="<extract.table.name>/<extract.extract.id>/<extract.type>"
writer.file.name.format="part.<writer_id>_batch_<dataset.batch_id>.<branch_id>.<format_extension>"
{code}
Notice the values (like "dataset.batch_id" comes from the runtime config(
state.getProp() ), so it allows you to have any kind of flexible path and file
name based on your use-case.
This will be enabled by the feature flag, so existing functionality can remains
the same.
was:
currently gobblin has hard coded specification for writer's path and file name
primarily it has namespace and tablename and default - 3 way to have writer's
path and file name.
{code:java}
// code placeholder
switch (getWriterFilePathType(state)) {
case NAMESPACE_TABLE:
// writer.file.path.format = <extract.namespace>/<extract.table.name>/
return getNamespaceTableWriterFilePath(state);
case TABLENAME:
// <extract.table.name>
return WriterUtils.getTableNameWriterFilePath(state);
default:
return WriterUtils.getDefaultWriterFilePath(state, numBranches, branchId);
}
{code}
Filename:
{code:java}
namespace.replaceAll("\\.", "/") + "/" + table + "/" + extractId + "_"
+ (isFull ? "full" : "append");
{code}
There is no way user can add any other parameters like version, batchId.
Also it would be awesome to have any configuration value to be part of the
writer path, which can be defined by the format like this
{code: java}
extract.type = increments
writer.file.path.format="<extract.table.name>/<extract.extract.id>/<extract.type>"
writer.file.name.format="part.<writer_id>_batch_<dataset.batch_id>.<branch_id>.<format_extension>"
{code}
This will be enabled by the feature flag, so existing functionality can remains
the same.
> provide a way to speficy writer path and file name format via config
> --------------------------------------------------------------------
>
> Key: GOBBLIN-1399
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1399
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-api
> Affects Versions: 0.15.0
> Reporter: Jay Sen
> Assignee: Hung Tran
> Priority: Major
> Fix For: 0.16.0
>
>
> currently gobblin has hard coded specification for writer's path and file name
> primarily it has namespace and tablename and default - 3 way to have writer's
> path and file name.
> {code:java}
> // code placeholder
> switch (getWriterFilePathType(state)) {
> case NAMESPACE_TABLE:
> // writer.file.path.format = <extract.namespace>/<extract.table.name>/
> return getNamespaceTableWriterFilePath(state);
> case TABLENAME:
> // <extract.table.name>
> return WriterUtils.getTableNameWriterFilePath(state);
> default:
> return WriterUtils.getDefaultWriterFilePath(state, numBranches, branchId);
> }
> {code}
>
> Filename:
> {code:java}
> namespace.replaceAll("\\.", "/") + "/" + table + "/" + extractId + "_"
> + (isFull ? "full" : "append");
> {code}
>
> There is no way user can add any other parameters like version, batchId.
> Also it would be awesome to have any configuration value to be part of the
> writer path, which can be defined by the format like this
>
> {code}
> Unable to find source-code formatter for language: java. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlextract.type = increments
> writer.file.path.format="<extract.table.name>/<extract.extract.id>/<extract.type>"
> writer.file.name.format="part.<writer_id>_batch_<dataset.batch_id>.<branch_id>.<format_extension>"
> {code}
> Notice the values (like "dataset.batch_id" comes from the runtime config(
> state.getProp() ), so it allows you to have any kind of flexible path and
> file name based on your use-case.
> This will be enabled by the feature flag, so existing functionality can
> remains the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
