Public bug reported:
I followed the solution in
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1908971 (I was
running duplicity 0.8.11.1612-1 as well), but after upgrading to 0.8.19,
the problem still persists for me when trying to use S3 multipart
uploads.
Note: This problem doesn't appear to exist in duplicity 0.7.17 running
on python2.7, which is making me wonder if this is a python3.8 related
issue.
user@host:~$ lsb_release -rd
Description: Ubuntu 20.04.2 LTS
Release: 20.04
user@host:~$ apt-cache policy duplicity
duplicity:
Installed: 0.8.19-ppa202104291804~ubuntu20.04.1
Candidate: 0.8.19-ppa202104291804~ubuntu20.04.1
Version table:
*** 0.8.19-ppa202104291804~ubuntu20.04.1 500
500
http://ppa.launchpad.net/duplicity-team/duplicity-release-git/ubuntu focal/main
amd64 Packages
100 /var/lib/dpkg/status
0.8.11.1612-1 500
500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/main amd64
Packages
Start duply v2.2, time is 2021-06-02 22:58:11.
Using profile '/root/.duply/artifactory_backup'.
Using installed duplicity version 0.8.19, python 3.8.5 (/usr/bin/python3), gpg
2.2.19 (Home: /root/.gnupg), awk 'GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU
MP 6.2.0)', grep 'grep (GNU grep) 3.4', bash '5.0.17(1)-release
(x86_64-pc-linux-gnu)'.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - En/Decryption skipped. (GPG disabled)
--- Start running command PRE at 22:58:11.550 ---
Skipping n/a script '/root/.duply/artifactory_backup/pre'.
--- Finished state OK at 22:58:11.563 - Runtime 00:00:00.013 ---
--- Start running command BKP at 22:58:11.595 ---
Using archive dir: /root/.cache/duplicity/duply_artifactory_backup
Using backup name: duply_artifactory_backup
GPG binary is gpg, version (2, 2, 19)
Import of duplicity.backends.adbackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.boxbackend Failed: No module named 'boxsdk'
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.gdrivebackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.idrivedbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.jottacloudbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.megav2backend Succeeded
Import of duplicity.backends.megav3backend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pcabackend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rclonebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.s3_boto3_backend Succeeded
Multiprocessing is not supported on linux, will use threads instead.
Import of duplicity.backends.s3_boto_backend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Setting multipart boto backend process pool to 4 processes
Reading globbing filelist /root/.duply/artifactory_backup/exclude
Main action: inc
Acquiring lockfile b'/root/.cache/duplicity/duply_artifactory_backup/lockfile'
================================================================================
duplicity 0.8.19
Args: /usr/bin/duplicity --name duply_artifactory_backup --no-encryption
--verbosity 9 --full-if-older-than 7D --volsize 1024 --s3-use-multiprocessing
--s3-multipart-chunk-size 512 --allow-source-mismatch --exclude-filelist
/root/.duply/artifactory_backup/exclude /MyDirectory
s3://s3-us-west-2.amazonaws.com/BUCKET-NAME/artifactory_backup
Linux ip-172-30-115-92 5.4.0-1047-aws #49-Ubuntu SMP Wed Apr 28 22:47:04 UTC
2021 x86_64 x86_64
/usr/bin/python3 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0]
===============
Log errors:
Attempt 4 failed. BackendException: Multipart upload failed. Aborted.
Writing duplicity-full.20210602T225820Z.vol1.difftar.gz
Uploading
s3://s3-us-west-2.amazonaws.com/BUCKET-NAME/artifactory_backup/duplicity-full.20210602T225820Z.vol1.difftar.gz
to STANDARD Storage
Uploading 1073705932 bytes in 2 chunks
Waiting for the pool to finish processing 2 tasks
Thread-29: Uploading chunk 1
Thread-30: Uploading chunk 2
Thread-30: Upload of chunk 2 failed. Retrying 4 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 4 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 3 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 3 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 2 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 2 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 1 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 1 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 0 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 0 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Aborting...
Thread-29: Upload of chunk 1 failed. Aborting...
Part upload not successful, aborting multipart upload.
A process pool already exists. Destroying previous pool.
Setting multipart boto backend process pool to 4 processes
Done waiting for the pool to finish processing
Backtrace of previous error: Traceback (innermost last):
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 384, in
inner_retry
return fn(self, *args)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 555, in put
self.__do_put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 541, in
__do_put
self.backend._put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_single.py",
line 265, in _put
self.upload(source_path.name, key, headers)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.py", line
192, in upload
raise BackendException(u"Multipart upload failed. Aborted.")
duplicity.errors.BackendException: Multipart upload failed. Aborted.
=========
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.py", line
223, in _upload
mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback,
File "/usr/lib/python3/dist-packages/boto/s3/multipart.py", line 257, in
upload_part_from_file
key.set_contents_from_file(fp, headers=headers, replace=replace,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 1307, in
set_contents_from_file
self.send_file(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 760, in send_file
self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 932, in
_send_file_internal
self.content_type = mimetypes.guess_type(self.path)[0]
File "/usr/lib/python3.8/mimetypes.py", line 292, in guess_type
return _db.guess_type(url, strict)
File "/usr/lib/python3.8/mimetypes.py", line 117, in guess_type
scheme, url = urllib.parse._splittype(url)
File "/usr/lib/python3.8/urllib/parse.py", line 1008, in _splittype
match = _typeprog.match(url)
TypeError: cannot use a string pattern on a bytes-like object
** Affects: duplicity (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Desktop
Packages, which is subscribed to duplicity in Ubuntu.
https://bugs.launchpad.net/bugs/1930640
Title:
Error when using S3 multipart upload - TypeError: cannot use a string
pattern on a bytes-like object
Status in duplicity package in Ubuntu:
New
Bug description:
I followed the solution in
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1908971 (I
was running duplicity 0.8.11.1612-1 as well), but after upgrading to
0.8.19, the problem still persists for me when trying to use S3
multipart uploads.
Note: This problem doesn't appear to exist in duplicity 0.7.17 running
on python2.7, which is making me wonder if this is a python3.8 related
issue.
user@host:~$ lsb_release -rd
Description: Ubuntu 20.04.2 LTS
Release: 20.04
user@host:~$ apt-cache policy duplicity
duplicity:
Installed: 0.8.19-ppa202104291804~ubuntu20.04.1
Candidate: 0.8.19-ppa202104291804~ubuntu20.04.1
Version table:
*** 0.8.19-ppa202104291804~ubuntu20.04.1 500
500
http://ppa.launchpad.net/duplicity-team/duplicity-release-git/ubuntu focal/main
amd64 Packages
100 /var/lib/dpkg/status
0.8.11.1612-1 500
500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal/main amd64
Packages
Start duply v2.2, time is 2021-06-02 22:58:11.
Using profile '/root/.duply/artifactory_backup'.
Using installed duplicity version 0.8.19, python 3.8.5 (/usr/bin/python3),
gpg 2.2.19 (Home: /root/.gnupg), awk 'GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2,
GNU MP 6.2.0)', grep 'grep (GNU grep) 3.4', bash '5.0.17(1)-release
(x86_64-pc-linux-gnu)'.
Checking TEMP_DIR '/tmp' is a folder and writable (OK)
Test - En/Decryption skipped. (GPG disabled)
--- Start running command PRE at 22:58:11.550 ---
Skipping n/a script '/root/.duply/artifactory_backup/pre'.
--- Finished state OK at 22:58:11.563 - Runtime 00:00:00.013 ---
--- Start running command BKP at 22:58:11.595 ---
Using archive dir: /root/.cache/duplicity/duply_artifactory_backup
Using backup name: duply_artifactory_backup
GPG binary is gpg, version (2, 2, 19)
Import of duplicity.backends.adbackend Succeeded
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.b2backend Succeeded
Import of duplicity.backends.boxbackend Failed: No module named 'boxsdk'
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.gdrivebackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.idrivedbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.jottacloudbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.mediafirebackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.megav2backend Succeeded
Import of duplicity.backends.megav3backend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pcabackend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rclonebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.s3_boto3_backend Succeeded
Multiprocessing is not supported on linux, will use threads instead.
Import of duplicity.backends.s3_boto_backend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Setting multipart boto backend process pool to 4 processes
Reading globbing filelist /root/.duply/artifactory_backup/exclude
Main action: inc
Acquiring lockfile b'/root/.cache/duplicity/duply_artifactory_backup/lockfile'
================================================================================
duplicity 0.8.19
Args: /usr/bin/duplicity --name duply_artifactory_backup --no-encryption
--verbosity 9 --full-if-older-than 7D --volsize 1024 --s3-use-multiprocessing
--s3-multipart-chunk-size 512 --allow-source-mismatch --exclude-filelist
/root/.duply/artifactory_backup/exclude /MyDirectory
s3://s3-us-west-2.amazonaws.com/BUCKET-NAME/artifactory_backup
Linux ip-172-30-115-92 5.4.0-1047-aws #49-Ubuntu SMP Wed Apr 28 22:47:04 UTC
2021 x86_64 x86_64
/usr/bin/python3 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0]
===============
Log errors:
Attempt 4 failed. BackendException: Multipart upload failed. Aborted.
Writing duplicity-full.20210602T225820Z.vol1.difftar.gz
Uploading
s3://s3-us-west-2.amazonaws.com/BUCKET-NAME/artifactory_backup/duplicity-full.20210602T225820Z.vol1.difftar.gz
to STANDARD Storage
Uploading 1073705932 bytes in 2 chunks
Waiting for the pool to finish processing 2 tasks
Thread-29: Uploading chunk 1
Thread-30: Uploading chunk 2
Thread-30: Upload of chunk 2 failed. Retrying 4 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 4 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 3 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 3 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 2 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 2 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 1 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 1 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Retrying 0 more times...
Thread-30: Uploading chunk 2
Thread-29: Upload of chunk 1 failed. Retrying 0 more times...
Thread-29: Uploading chunk 1
Thread-30: Upload of chunk 2 failed. Aborting...
Thread-29: Upload of chunk 1 failed. Aborting...
Part upload not successful, aborting multipart upload.
A process pool already exists. Destroying previous pool.
Setting multipart boto backend process pool to 4 processes
Done waiting for the pool to finish processing
Backtrace of previous error: Traceback (innermost last):
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 384, in
inner_retry
return fn(self, *args)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 555, in put
self.__do_put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 541, in
__do_put
self.backend._put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_single.py",
line 265, in _put
self.upload(source_path.name, key, headers)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.py",
line 192, in upload
raise BackendException(u"Multipart upload failed. Aborted.")
duplicity.errors.BackendException: Multipart upload failed. Aborted.
=========
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.py",
line 223, in _upload
mp.upload_part_from_file(fd, offset + 1, cb=_upload_callback,
File "/usr/lib/python3/dist-packages/boto/s3/multipart.py", line 257, in
upload_part_from_file
key.set_contents_from_file(fp, headers=headers, replace=replace,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 1307, in
set_contents_from_file
self.send_file(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 760, in send_file
self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/lib/python3/dist-packages/boto/s3/key.py", line 932, in
_send_file_internal
self.content_type = mimetypes.guess_type(self.path)[0]
File "/usr/lib/python3.8/mimetypes.py", line 292, in guess_type
return _db.guess_type(url, strict)
File "/usr/lib/python3.8/mimetypes.py", line 117, in guess_type
scheme, url = urllib.parse._splittype(url)
File "/usr/lib/python3.8/urllib/parse.py", line 1008, in _splittype
match = _typeprog.match(url)
TypeError: cannot use a string pattern on a bytes-like object
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1930640/+subscriptions
--
Mailing list: https://launchpad.net/~desktop-packages
Post to : desktop-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~desktop-packages
More help : https://help.launchpad.net/ListHelp
