Currently, locality groups can not be configured for AccumuloFileoutputFormat, it can only write to the default locality group. However things can still work out nicely, because the default locality group will track up to a 1000 column families. It will use these tracked column families at scan time to determine if the file should be used. The following is an example of this.
* Locality Group A has families 3, 4, 5 * Locality Group B has families x, y, z * Spark job 1 writes to Rfile RF1 families y,z. This data will end up in default LG. RF1 is imported to tablet T1. * Spark job 2 writes to Rfile RF2 families 3,5. This data will end up in default LG. RF2 is imported.to tablet T1. * A scan comes in for tablet T1 for family 3. The scan will examine RF1 and RF2 family metadata and only use RF2. * If T1 compacts RF1 and RF2 into a single file RF3, that file will have two locality groups. When the compaction reads LG B it will only read data from RF1. When it reads LG A it will only read data from RF2. On Fri, Oct 28, 2016 at 9:37 AM, Mario Pastorelli <[email protected]> wrote: > Hi, > > I have a question about using bulk ingestion for a rather special case. > Let's say that I have the locality groups A and B. The values of each > locality group are written to Accumulo in at different times, which means > that first we ingest all the cells of the group A and then of B. We use > Spark to ingest those records. Right now we write all the values with a > custom writer but we would like to create the rfiles directly with Spark. In > the case above, we would have two jobs creating the rfiles for the two > distinct locality groups. Is Accumulo able to import these files, > considering that they are two different locality groups, without triggering > a huge major compaction? If not, what strategy would you suggest for the > above use case? > > Thanks, > Mario > > -- > Mario Pastorelli | TERALYTICS > > software engineer > > Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland > phone: +41794381682 > email: [email protected] > www.teralytics.net > > Company registration number: CH-020.3.037.709-7 | Trade register Canton > Zurich > Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann > de Vries > > This e-mail message contains confidential information which is for the sole > attention and use of the intended recipient. Please notify us at once if you > think that it may not be intended for you and delete it immediately.
