worker_limit reached for map reduce jobs

[Girish Shankarraman]

Hello,

Currently using Riak 2.1.1. I have 7 nodes in my cluster.
I am looking for some information on how these worker_limits are configured .


I have 50 linux(client) hosts trying to run the map reduce jobs on Riak. I am 
getting the error below where some of the hosts complain about the worker_limit 
being reached.

Looking for some insights on whether I can tune the system to avoid this error? 
Couldn't find too much documentation around the worker_limit.

{"phase":0,"error":"[worker_limit_reached]","input":"{<<\"provisionentry\">>,<<\"R89Okhz49SDje0y0qvcnkK7xLH0\">>}","type":"result","stack":"[]"}
 with query MapReduce(path='/mapred', reply_headers={'content-length': '144', 
'access-control-allow-headers': 'Content-Type', 'server': 'MochiWeb/1.1 
WebMachine/1.10.8 (that head fake, tho)', 'connection': 'close', 'date': 'Thu, 
27 Aug 2015 00:32:22 GMT', 'access-control-allow-origin': '*', 
'access-control-allow-methods': 'POST, GET, OPTIONS', 'content-type': 
'application/json'}, verb='POST', headers={'Content-Type': 'application/json'}

Thanks,

- Girish Shankarraman

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

indexes_not_supported,riak_kv_bitcask_backend error on multi backend with leveldb

[Bora Kou]

Hi,

I am testing out riak kv multi backend with leveldb and bitcask.  I
configured bitcask as the default backend and leveldb with bucket-type.  I
am using riak kv version 2.0.5-1.

I am able to do http PUT with index, but when I try to GET with the index,
it give me "{error,{indexes_not_supported,riak_kv_bitcask_backend}}}"

Am I missing something?  Is this type of setting supported?

Here is the information:

## PUT (with index work fine)
*curl -v -XPUT
localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/keys/john \*
*  -H "x-riak-index-email_bin: j...@example.com " \*
*  -d 'John Testing'*

** About to connect() to localhost port 8098 (#0)*
**   Trying 127.0.0.1... connected*
*> PUT /types/xfiles/buckets/xfiles_master_aws_pod98/keys/john HTTP/1.1*
*> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
OpenSSL/1.0.1 zlib/1.2.3.4  libidn/1.23 librtmp/2.3*
*> Host: localhost:8098*
*> Accept: */**
*> x-riak-index-email_bin: j...@example.com *
*> Content-Length: 12*
*> Content-Type: application/x-www-form-urlencoded*
*> *
** upload completely sent off: 12out of 12 bytes*
*< HTTP/1.1 204 No Content*
*< Vary: Accept-Encoding*
*< Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)*
*< Date: Thu, 27 Aug 2015 23:32:20 GMT*
*< Content-Type: application/x-www-form-urlencoded*
*< Content-Length: 0*
*< *
** Connection #0 to host localhost left intact*
** Closing connection #0*

## GET INDEX (got error)
*curl -v
localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/index/email_bin/j...@example.com
*

** About to connect() to localhost port 8098 (#0)*
**   Trying 127.0.0.1... connected*
*> GET
/types/xfiles/buckets/xfiles_master_aws_pod98/index/email_bin/j...@example.com
 HTTP/1.1*
*> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
OpenSSL/1.0.1 zlib/1.2.3.4  libidn/1.23 librtmp/2.3*
*> Host: localhost:8098*
*> Accept: */**
*> *
*< HTTP/1.1 500 Internal Server Error*
*< Vary: Accept-Encoding*
*< Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)*
*< Date: Thu, 27 Aug 2015 23:41:52 GMT*
*< Content-Type: text/html*
*< Content-Length: 305*
*< *
** Connection #0 to host localhost left intact*
** Closing connection #0*
*500 Internal Server
ErrorInternal Server ErrorThe server
encountered an error while processing this
request:{error,{error,{indexes_not_supported,riak_kv_bitcask_backend}}}mochiweb+webmachine
web server*

## GET THE KEY (work fine)
*curl localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/keys/john*
*John Testing*


## BUCKET-TYPE
*riak-admin bucket-type list*
*default (active)*
*xfiles (active)*

*riak-admin bucket-type status xfiles*
*xfiles is active*

*active: true*
*allow_mult: true*
*backend: <<"leveldb_multi">>*
*basic_quorum: false*
*big_vclock: 50*
*chash_keyfun: {riak_core_util,chash_std_keyfun}*
*claimant: 'r...@riak-test1.example.com '*
*dvv_enabled: true*
*dw: quorum*
*last_write_wins: false*
*linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}*
*n_val: 3*
*notfound_ok: true*
*old_vclock: 86400*
*postcommit: []*
*pr: 0*
*precommit: []*
*pw: 0*
*r: quorum*
*rw: quorum*
*small_vclock: 50*
*w: quorum*
*young_vclock: 20*


## MY CONFIGURATION (6 nodes)
*cat /etc/riak/riak.conf | grep multi*
*storage_backend = multi*
*multi_backend.bitcask_multi.storage_backend = bitcask*
*multi_backend.bitcask_multi.bitcask.data_root = /var/lib/riak/bitcask*
*multi_backend.leveldb_multi.storage_backend = leveldb*
*multi_backend.leveldb_multi.leveldb.data_root = /var/lib/riak/leveldb*
*multi_backend.leveldb_multi.leveldb.maximum_memory.percent = 30*
*multi_backend.leveldb_multi.leveldb.compaction.trigger.tombstone_count =
1000*
*multi_backend.default = bitcask_multi*

*cat /var/lib/riak/generated.configs/* | grep multi*
*  {storage_backend,riak_kv_multi_backend},*
*  {multi_backend_default,<<"bitcask_multi">>},*
*  {multi_backend,*
*  [{<<"bitcask_multi">>,riak_kv_bitcask_backend,*
*   {<<"leveldb_multi">>,riak_kv_eleveldb_backend,*




Thank you,


-Bora
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: indexes_not_supported,riak_kv_bitcask_backend error on multi backend with leveldb

[Bora Kou]

We found the issue.  It turned out that the "storage_backend" setting is
still bitcask on all the node except the one I was testing on.  After
making sure all node have "storage_backend = multi", the secondary index
works as expected.  My bad, sorry for the noise :(

-Bora

On Thu, Aug 27, 2015 at 5:05 PM, Bora Kou  wrote:

> Hi,
>
> I am testing out riak kv multi backend with leveldb and bitcask.  I
> configured bitcask as the default backend and leveldb with bucket-type.  I
> am using riak kv version 2.0.5-1.
>
> I am able to do http PUT with index, but when I try to GET with the index,
> it give me "{error,{indexes_not_supported,riak_kv_bitcask_backend}}}"
>
> Am I missing something?  Is this type of setting supported?
>
> Here is the information:
>
> ## PUT (with index work fine)
> *curl -v -XPUT
> localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/keys/john \*
> *  -H "x-riak-index-email_bin: j...@example.com " \*
> *  -d 'John Testing'*
>
> ** About to connect() to localhost port 8098 (#0)*
> **   Trying 127.0.0.1... connected*
> *> PUT /types/xfiles/buckets/xfiles_master_aws_pod98/keys/john HTTP/1.1*
> *> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
> OpenSSL/1.0.1 zlib/1.2.3.4  libidn/1.23 librtmp/2.3*
> *> Host: localhost:8098*
> *> Accept: */**
> *> x-riak-index-email_bin: j...@example.com *
> *> Content-Length: 12*
> *> Content-Type: application/x-www-form-urlencoded*
> *> *
> ** upload completely sent off: 12out of 12 bytes*
> *< HTTP/1.1 204 No Content*
> *< Vary: Accept-Encoding*
> *< Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)*
> *< Date: Thu, 27 Aug 2015 23:32:20 GMT*
> *< Content-Type: application/x-www-form-urlencoded*
> *< Content-Length: 0*
> *< *
> ** Connection #0 to host localhost left intact*
> ** Closing connection #0*
>
> ## GET INDEX (got error)
> *curl -v
> localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/index/email_bin/j...@example.com
> *
>
> ** About to connect() to localhost port 8098 (#0)*
> **   Trying 127.0.0.1... connected*
> *> GET
> /types/xfiles/buckets/xfiles_master_aws_pod98/index/email_bin/j...@example.com
>  HTTP/1.1*
> *> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
> OpenSSL/1.0.1 zlib/1.2.3.4  libidn/1.23 librtmp/2.3*
> *> Host: localhost:8098*
> *> Accept: */**
> *> *
> *< HTTP/1.1 500 Internal Server Error*
> *< Vary: Accept-Encoding*
> *< Server: MochiWeb/1.1 WebMachine/1.10.5 (jokes are better explained)*
> *< Date: Thu, 27 Aug 2015 23:41:52 GMT*
> *< Content-Type: text/html*
> *< Content-Length: 305*
> *< *
> ** Connection #0 to host localhost left intact*
> ** Closing connection #0*
> *500 Internal Server
> ErrorInternal Server ErrorThe server
> encountered an error while processing this
> request:{error,{error,{indexes_not_supported,riak_kv_bitcask_backend}}}mochiweb+webmachine
> web server*
>
> ## GET THE KEY (work fine)
> *curl
> localhost:8098/types/xfiles/buckets/xfiles_master_aws_pod98/keys/john*
> *John Testing*
>
>
> ## BUCKET-TYPE
> *riak-admin bucket-type list*
> *default (active)*
> *xfiles (active)*
>
> *riak-admin bucket-type status xfiles*
> *xfiles is active*
>
> *active: true*
> *allow_mult: true*
> *backend: <<"leveldb_multi">>*
> *basic_quorum: false*
> *big_vclock: 50*
> *chash_keyfun: {riak_core_util,chash_std_keyfun}*
> *claimant: 'r...@riak-test1.example.com '*
> *dvv_enabled: true*
> *dw: quorum*
> *last_write_wins: false*
> *linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}*
> *n_val: 3*
> *notfound_ok: true*
> *old_vclock: 86400*
> *postcommit: []*
> *pr: 0*
> *precommit: []*
> *pw: 0*
> *r: quorum*
> *rw: quorum*
> *small_vclock: 50*
> *w: quorum*
> *young_vclock: 20*
>
>
> ## MY CONFIGURATION (6 nodes)
> *cat /etc/riak/riak.conf | grep multi*
> *storage_backend = multi*
> *multi_backend.bitcask_multi.storage_backend = bitcask*
> *multi_backend.bitcask_multi.bitcask.data_root = /var/lib/riak/bitcask*
> *multi_backend.leveldb_multi.storage_backend = leveldb*
> *multi_backend.leveldb_multi.leveldb.data_root = /var/lib/riak/leveldb*
> *multi_backend.leveldb_multi.leveldb.maximum_memory.percent = 30*
> *multi_backend.leveldb_multi.leveldb.compaction.trigger.tombstone_count =
> 1000*
> *multi_backend.default = bitcask_multi*
>
> *cat /var/lib/riak/generated.configs/* | grep multi*
> *  {storage_backend,riak_kv_multi_backend},*
> *  {multi_backend_default,<<"bitcask_multi">>},*
> *  {multi_backend,*
> *  [{<<"bitcask_multi">>,riak_kv_bitcask_backend,*
> *   {<<"leveldb_multi">>,riak_kv_eleveldb_backend,*
>
>
>
>
> Thank you,
>
>
> -Bora
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak Go Client 1.1 is Available!

[Alex Moore]

Hi All,

The Go client for Riak is out of beta, so Go get it!

To install it in your go project, just run `go get 
github.com/basho/riak-go-client` .

You can find more information on the project page 
(https://github.com/basho/riak-go-client 
), and in the API docs 
(https://godoc.org/github.com/basho/riak-go-client 
).

Special thanks to Luke Bakkan & Chris Mancini for their work on the client, 
Timo Gatsonides for his inspiring work on goriakpbc, and Sergio Arteaga for 
submitting GitHub issues.

Thanks,
Alex


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com