[
https://issues.apache.org/jira/browse/GORA-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485370#comment-13485370
]
raf shin edited comment on GORA-173 at 10/27/12 7:13 AM:
---------------------------------------------------------
We have changed Nutch so that writing fetched pages is done in separate threads
than that of hadoop or fetchers. so we have created a store for each of our
writer threads. The writer threads just create the store and call
store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have
multiple threads on one store.
We have been using the same code from 6 months ago but we didn't face this
exception until recently that we have increased our bandwidth from 10Mbps to
100Mbps.
private class WriteThread extends Thread {
private DataStore<String, WebPage> storeWebPage = null;
public WriteThread(){
storeWebPage = StorageUtils.createWebStore(conf, String.class,
WebPage.class);
....
}
public void run(){
...
fit.page.clearDirty("content");
extractOutlinks(fit);
updateWebPage(fit.url, fit.page);
storeWebPage.put(key, fit.page);
...
}
was (Author: rafshin):
We have changed Nutch so that writing fetched pages is done in separate
threads than that of hadoop or fetchers. so we have created a store for each of
our writer threads. The writer threads just create the store and call
store.put(key, webpage).
Because we have a store for each writer thread it is not the case that we have
multiple threads on one store.
We have been using the same code from 6 months ago but we didn't face this
exception until recently that we have increased our bandwidth from 10Mbps to
100Mbps.
> Iterating over StatefulHashMap in HBaseStore throws
> ConcurrentModificationException
> -----------------------------------------------------------------------------------
>
> Key: GORA-173
> URL: https://issues.apache.org/jira/browse/GORA-173
> Project: Apache Gora
> Issue Type: Bug
> Affects Versions: 0.1.1-incubating
> Reporter: raf shin
>
> We are using just one thread working with HBaseStore in Nutch.
> HBaseStore.put throws the following exception. I think the same problem was
> in CassandraStore that is
> solved now. But the problem in HBaseStore remains. We used the same solution
> for HBaseStore but
> it didn't work.
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at org.apache.gora.hbase.store.HBaseStore.put(HBaseStore.java:267)
> at
> org.nise.hiva.fetcher.FetcherReducer$WriteThread.run(FetcherReducer.java:1557)
> Line 266 in HBaseStore:
> StatefulHashMap<Utf8, ?> map = (StatefulHashMap<Utf8, ?>) o;
> for (Entry<Utf8, State> e : map.states().entrySet()) {
> Utf8 mapKey = e.getKey();
> switch (e.getValue()) {
> case DIRTY:
> byte[] qual = Bytes.toBytes(mapKey.toString());
> byte[] val = toBytes(map.get(mapKey),
> field.schema().getValueType());
> put.add(hcol.getFamily(), qual, val);
> hasPuts = true;
> break;
> case DELETED:
> qual = Bytes.toBytes(mapKey.toString());
> hasDeletes = true;
> delete.deleteColumn(hcol.getFamily(), qual);
> break;
> }
> ***************************************************************************
> Our solution was :
> Set<Utf8> keys = map.keySet();
> Utf8 [] arr = (Utf8[])keys.toArray();
> for(Utf8 key: arr){....}
> But it didn't work and we throw the same exception.
> Our version is 0.1 but i checked the trunk and did't see much change in that
> code so the trunk code
> may throw the same exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
