Re: Download stops at certain megabyte boundary (RiakCS)
Martin, I think you have already read this page: http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/#Setting-up-the-Proper-Riak-Backend As it says, you can use add_paths to set load path. -- Shunichi Shinohara Basho Japan KK On Wed, Sep 11, 2013 at 2:05 AM, Martin Alpers martin-alp...@web.de wrote: Thank you Shino for your patient assistance. I did follow Seth's advice, and upgrading resulted in smooth downloads, as stated in my reply to his mail. Additionally, I upgraded the one noed I am working on to the very recent 1.4.1 version. (Riak is 1.4.2) I still use the packages for Debian Squeeze. Outputs of console.log and crash.log are attached to this email. Although I am not an expert with these, the term invalid_storage_backend appears in both of them, and riak used to start just fine before. I also tried to find the files: root@riak2:~# find /usr/lib/riak -name *kv_multi* /usr/lib/riak/lib/riak_kv-1.4.2-0-g61ac9d8/ebin/riak_kv_multi_backend.beam root@riak2:~# find /usr/lib/riak-cs -name *kv_multi* /usr/lib/riak-cs/lib/riak_cs-1.4.1/ebin/riak_cs_kv_multi_backend.beam So is adding -pz /usr/lib/riak-cs/lib/riak_cs-1.4.1/ebin/ to /etc/riak/vm.args, as suggested here: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-October/002211.html the right thing to do? Kind regards, Martin On 13/09/10.10:48:1378806537, Shunichi Shinohara wrote: Martin, Thank you for more information. I found a mis-configuration of riak's app.config. The backend of riak for Riak CS is riak_cs_kv_multi_backend (look out for _cs_). This is the custom riak_kv backend for Riak CS. The error of s3cmd ls was caused by this. As for MB boundary stop, the issue https://github.com/basho/riak_cs/issues/613 may be related. Although not direct way to treat it, some points which may help: - As Seth said, upgrade to 1.4, which can reduce network communication between riak nodes - Set pb_backlog to 256 or higher in riak_api section of riak's app.config Regards, Shino ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Greetings, Martin Alpers -- martin-alp...@web.de Mobile: 0176/66185173, but I prefer typing to talking (: Jabber: martin.alp...@jabber.org My mails are signed using GPG to verify their origin; request my public key (10216CFB). See also: http://apps.opendatacity.de/stasi-vs-nsa/ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Download stops at certain megabyte boundary (RiakCS)
Martin, Thank you for more information. I found a mis-configuration of riak's app.config. The backend of riak for Riak CS is riak_cs_kv_multi_backend (look out for _cs_). This is the custom riak_kv backend for Riak CS. The error of s3cmd ls was caused by this. As for MB boundary stop, the issue https://github.com/basho/riak_cs/issues/613 may be related. Although not direct way to treat it, some points which may help: - As Seth said, upgrade to 1.4, which can reduce network communication between riak nodes - Set pb_backlog to 256 or higher in riak_api section of riak's app.config Regards, Shino ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Download stops at certain megabyte boundary (RiakCS)
Martin,This may not be a silver bullet for this issue but as far as version compatibility between Riak and Riak CS goes we have a chart here:http://docs.basho.com/riakcs/latest/cookbooks/Version-Compatibility/I'd highly recommend upgrading your Riak CS as there are some known issues with the 1.3.x series running against Riak 1.4.x --Seth Thomas On September 6, 2013 at 1:08:43 PM, Martin Alpers (martin-alp...@web.de) wrote: Thanks Shino for your reply.
I could slap myself for failing to look at the logs at all.
Altough I did not get a clue from this anyway, a nonempty crashlog is in itself a hint.
* downloading from the frist node stopped at exactly 3MB, i.e. 3145728 bytes
* downloading from the second node stopped at exactly 22MB
[snip]
I also tried your URL and found sudden stop at MB boundary.
Could you please add some information:
- Riak version
1.4.1-1
- Riak CS version
1.3.1-1
I read a few days ago a new version is available.
Could I just upgrade or do I have do disassemble the cluster, or anything of that kind?
- app.config and vm.args of Riak
Both files from the first attached with this mail, in the archive config.tar.gz
Respective files from other nodes differ only in two IPs and -name, says GNU diff.
- app.config and vm.args of Riak CS
See my comments on the Riak files
- log of Riak and Riak CS if there is output around download
I created a tar archive with messages from Riak CS only because Riak itself did not generate any messages around that time.
error.log is not included because each line there is found in console.log as well. (I diff'ed them.)
(I trimmed surrounding messages generated by the garbage collector once every 15 minutes, severity "info".)
See attachment get-logs.tar.gz
I have another problem that might or might not be related, and is described much easier:
s3cmd ls s3://abc
WARNING: Retrying failed request: /?delimiter=/ ('')
Log might be useful also for this issue.
Again console.log and error.log differ only in garbage collection lines, so I will include error.log.
In this case, both console.log and error.log are appended two lines; they are saved in riak.log.
See attachment mb-logs.tar.gz
One remaining question:
root@riak3:~# riak-admin diag
...
["sh: 1: exec: sysctl: not found"],
It may be better to separate this from above ones ;)
Would you file this issue at https://github.com/basho/riak/issues ?
Certainly. Thanks for the hint.
--
Greetings, Martin Alpers
--
martin-alp...@web.de
Mobile: 0176/66185173, but I prefer typing to talking (:
Jabber: martin.alp...@jabber.org
My mails are signed using GPG to verify their origin; request my public key (10216CFB).
See also: http://apps.opendatacity.de/stasi-vs-nsa/
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Download stops at certain megabyte boundary (RiakCS)
Hello everyone on the list,
I run a test cluster for RiakCS, running on three nodes, mostly using defaults.
Mostly meaning that all modifications were necessary to make RiakCS run, but no
tweaks have taken place yet.
Now after figuring out how to make objects publicly available we tried to
download it again, but we saw a strange behavior:
* downloading from the frist node stopped at exactly 3MB, i.e. 3145728 bytes
* downloading from the second node stopped at exactly 22MB
* downloading from the third node stopped at exactly 16MB
* downloading from the first node using s3cmd stopped at exactly 2MB
* downloading from the first node using wget stopped several times precisely on
MB boundaries, but continued afterwards until the download was complete
All halts are abrupt, a 2Mb download slows down to 0b in an instant.
The first two cases were exactly the same for a friend of mine, the remaining
were only tested by myself.
If you want to test that for yourself, the file can be found at
http://85.10.193.103:6080$x/xyz/spying_nsa.webm, where $x is the number of the
node.
When I first uploaded the file, I interupted c3cmd (Crtl-C) because I had
forgotten to allow it to be read publicly, and then reuploaded it.
However, I uploaded the file again to another bucket, available at at
.../abc/sn.webm, and it shows the same behavior.
The number might be different, but the third node is unresponsive as well.
Naturally I had checked if the processes are still up, and riak-admin
member-status, and did not find any peculiarities.
I read in the docs about the enable_multipart setting in .s3cfg, but mine is
False and uploading a file of 25MB worked nicely.
(See
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-an-S3-Client/)
My .s3cfg was generated by s3cmd and the only modified settings are those for
login and proxy settings.
For reference, the file, redacted only to remove credentials, can be found as
an attachment to this mail.
Now what could I have done wrong?
I had several configuration errors in the /etc/riak-cs/app.configs of the
cluster, fixed them, and restarted riak-cs, but the problem persists.
Could it be a setting in .s3cfg?
What about ulimit? It is still defaulting to 1024, but this is a cluster
storing a mere 500MB or so, and less than 30 files, serving a few objects per
hour.
I mean, what would that number look like on a production cluster if 1024 is too
little for such low-profile testing?
For the same reason, I am sure that IO is not the problem - my 2Mb ADSL will be
the bottleneck, and 50kb upload even more so.
I have another problem that might or might not be related, and is described
much easier:
s3cmd ls s3://abc
WARNING: Retrying failed request: /?delimiter=/ ('')
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?delimiter=/ ('')
WARNING: Waiting 6 sec...
^CSee ya!
Happens when I try to list objects of a non-empty bucket. Any ideas on that one?
One remaining question:
root@riak3:~# riak-admin diag
...
[sh: 1: exec: sysctl: not found],
...
root@riak3:~#
I know how this message comes about:
sysctl lives under /sbin, which is not in $PATH for any user except root.
Should I care? What is the best way to fix that? I can think of a few.
Platform information:
Three instances of KVM, running Debian 7 (Squeeze), each connected to the KVM
host using routed TAP devices
Erlang: Version: 1:15.b.1-dfsg-4 (R15B01 is needed to compile Riak, so this
looks fine for me)
Prebuilt riak and riak-cs packages built for Debian 6 (Squeeze)
To meet dependencies, libssl0.9.8 from Squeeze was installed using dpkg -i
(Wheezy only has libssl1.0.0, which does not satisfy said dependency.)
Big thanks to anyone who has a look at my problems.
Any help is highly appreciated and I will gladly provide any information that I
failed to provide in the first place.
--
Greetings, Martin Alpers
--
martin-alp...@web.de
Mobile: 0176/66185173, but I prefer typing to talking (:
Jabber: martin.alp...@jabber.org
My mails are signed using GPG to verify their origin; request my public key
(10216CFB).
See also: http://apps.opendatacity.de/stasi-vs-nsa/
[default]
bucket_location = DE
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
enable_multipart = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes
--passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes
--passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
invalidate_on_cf = False
list_md5 = False
log_target_prefix =
mime_type =
multipart_chunk_size_mb = 15
preserve_attrs = True
Re: Download stops at certain megabyte boundary (RiakCS)
Hi Martin,
Thank you for sharing detailed information.
On Wed, Sep 4, 2013 at 11:20 PM, Martin Alpers martin-alp...@web.de wrote:
* downloading from the frist node stopped at exactly 3MB, i.e. 3145728 bytes
* downloading from the second node stopped at exactly 22MB
[snip]
I also tried your URL and found sudden stop at MB boundary.
Could you please add some information:
- Riak version
- Riak CS version
- app.config and vm.args of Riak
- app.config and vm.args of Riak CS
- log of Riak and Riak CS if there is output around download
# If log files are large, please send them to me (sh...@basho.com).
I have another problem that might or might not be related, and is described
much easier:
s3cmd ls s3://abc
WARNING: Retrying failed request: /?delimiter=/ ('')
Log might be useful also for this issue.
One remaining question:
root@riak3:~# riak-admin diag
...
[sh: 1: exec: sysctl: not found],
It may be better to separate this from above ones ;)
Would you file this issue at https://github.com/basho/riak/issues ?
Regards,
Shino
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
