It would be similar to case 2 right. Say for example in spark I read a file of size 512mb which would span 4 cores(if block size is 128). Executor will be spanned based on data locality and if the executor is launched machine1,2,3,4. May be if block3 region is handled by machine6, then when I bulk load via spark hbase connector (which uses saveAsNewHadoopApidataset) then in this case the HFILE write of block3 would be to any of those 4 machines and not to machine6. Is that right? Or i misunderstood?
On Jan 22, 2018 22:27, "Ted Yu" <yuzhih...@gmail.com> wrote: > For case 1, HFile would be loaded into the region (via staging directory). > > Please see: > http://hbase.apache.org/book.html#arch.bulk.load > > On Mon, Jan 22, 2018 at 8:52 AM, vignesh <vignesh...@gmail.com> wrote: > > > If it is a bulk load I use spark hbase connector provided by hortonworks. > > For time series writes I use normal hbase client API's. > > > > So does that mean in case 2(client API write) the write to memstore will > > happen via network? In case 1(bulk load)the HFile will be moved to the > > region which is supposed to hold or it will write to local and keep that > as > > a copy and the second replication would go to that particular region? > > > > On Jan 22, 2018 22:16, "Ted Yu" <yuzhih...@gmail.com> wrote: > > > > Which connector do you use to perform the write ? > > > > bq. Or spark will wisely launch an executor on that machine > > > > I don't think that is the case. Multiple writes may be performed which > > would end up on different region servers. Spark won't provide the > affinity > > described above. > > > > On Mon, Jan 22, 2018 at 7:19 AM, vignesh <vignesh...@gmail.com> wrote: > > > > > Hi, > > > > > > I have a Spark job which reads some timeseries data and pushes that to > > > HBASE using HBASE client API. I am executing this Spark job on a 10 > > > node cluster. Say at first when spark kicks off it picks > > > machine1,machine2,machine3 as its executors. Now when the job inserts > > > a row to HBASE. Below is what my undersatnding on what it does. > > > > > > Based on the row key a particular region(from the META) would be > > > chosen and that row will be pushed to that RegionServer's memstore and > > > WAL and once the memestore is full it would be flushed to the disk.Now > > > if assume a particular row is being processed by a executor on > > > machine2 and the regionserver which handles that region to which the > > > put is to be made is on machine6. Will the data be transferred from > > > machine2 to machine6 over network and then the data will be stored in > > > memstore of machine6. Or spark will wisely launch an executor on that > > > machine during write(if the dynamic allocation is turned on) and > > > pushes to it? > > > > > > > > > -- > > > I.VIGNESH > > > > > >
