Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/273#discussion_r63321660
--- Diff:
library/src/main/java/com/datatorrent/lib/io/fs/AbstractSingleFileOutputOperator.java
---
@@ -39,10 +41,37 @@
@NotNull
protected String outputFileName;
+ /**
+ * partitionedFileName string format specifier
+ e.g. fileName_physicalPartionId -> %s_%d
+ */
+ private String partitionedFileNameformat = "%s_%d";
+
+ /**
+ * Derived name for file based on physicalPartitionId
+ */
+ private transient String partitionedFileName;
+
+ /**
+ * Physical partition id for the current partition.
+ */
+ private transient int physicalPartitionId;
+
+ /**
+ * Initializing current partition id, partitionedFileName etc.
{@inheritDoc}
+ */
+ @Override
+ public void setup(OperatorContext context)
--- End diff --
My assumption is : if we have multiple partitions then each partition must
write to different file. Thus giving this choice to the end use may not be good
idea.
Regarding adding operatorId suffix only in case when there are more than
one partitions. This is good suggestion, but there is no direct way of getting
number of physical instances for the operator in the setup method. We might
need to add custom partitioner for doing this. IMO, it is over-complicating the
thing for the delta value addition.
Can you think of simpler way of knowing no. of physical partitions? OR do
you feel that it should be OK without worrying too much about it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
