You can add those properties in your super class and simply cast the clone to that class, so change:
*AbstractFileInputOperator<String> oper = cloneObject(kryo, this);* to something like: MyFileInputOperator<String> oper = (MyFileInputOperator) cloneObject(kryo, this); On Tue, Jun 28, 2016 at 11:17 AM, Mukkamula, Suryavamshivardhan (CWM-NR) < [email protected]> wrote: > Hi Ram, > > > > I would like to add the parameters for each partition like below. Each > operator to be given with its own configuration file and source identifier. > If there is any other way please let me know ? > > In my current definepartition() method , I am doing similarly like below, > but I have to add setter and getter methods in AbstractFileInputOperator > class. > > > > for (int j = 0; j < numDirs; ++j) { > > int first = sliceFirstIndex[j]; > > int last = first + partitionCounts[j]; > > String dir = directories[j]; > > LOG.info("definePartitions: first = {}, last = {}, dir = {}", first, > last, dir); > > for (int i = first; i < last; ++i) { > > AbstractFileInputOperator<String> oper = cloneObject(kryo, this); > > oper.setDirectory(dir); > > oper.setSourceId(<sourceId>); > > oper.setConfigFile(<fileName>); > > //oper.setpIndex(i); > > SlicedDirectoryScanner scn = (SlicedDirectoryScanner) > scanners.get(i); > > scn.setStartIndex(first); > > scn.setEndIndex(last); > > scn.setDirectory(dir); > > > > oper.setScanner(scn); > > newPartitions.add(new DefaultPartition<>(oper)); > > newManagers.add(oper.getIdempotentStorageManager()); > > } > > } > > > > > > Regards, > > Surya Vamshi > > > > *From:* Munagala Ramanath [mailto:[email protected]] > *Sent:* 2016, June, 28 2:03 PM > *To:* [email protected] > *Subject:* Re: Reading Multiple directories in parallel > > > > Not sure I fully understand the question but you can add whatever fields > you need > > to your class that extends *AbstractFileInputOperator*. For example, > > > https://github.com/DataTorrent/examples/blob/master/tutorials/fileIO-multiDir/src/main/java/com/example/fileIO/FileReaderMultiDir.java > > defines fields *directories* and *partitionCounts*. > > > > You can then set these fields as needed in *definePartitions*. > > > > Ram > > > > On Tue, Jun 28, 2016 at 10:31 AM, Mukkamula, Suryavamshivardhan (CWM-NR) < > [email protected]> wrote: > > Hi Ram, > > > > Can you please suggest , how would I add another variable (like > ‘directory’) while creating multiple partitions of > AbstractFileInputOperator in the define partition method. > > > > I have currently added variables in the AbstractFileInputOperator , which > I guess not a better way. > > > > These variables are basically used to scan directories in parallel > differently. > > > > Regards, > > Surya Vamshi > > > > _______________________________________________________________________ > > If you received this email in error, please advise the sender (by return > email or otherwise) immediately. You have consented to receive the attached > electronically at the above-noted email address; please retain a copy of > this confirmation for future reference. > > Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur > immédiatement, par retour de courriel ou par un autre moyen. Vous avez > accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à > l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de > cette confirmation pour les fins de reference future. > > > > _______________________________________________________________________ > > If you received this email in error, please advise the sender (by return > email or otherwise) immediately. You have consented to receive the attached > electronically at the above-noted email address; please retain a copy of > this confirmation for future reference. > > Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur > immédiatement, par retour de courriel ou par un autre moyen. Vous avez > accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à > l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de > cette confirmation pour les fins de reference future. > >
