Pydantic (https://pydantic-docs.helpmanual.io/) can already do that.
On Monday, 13 July 2020 21:03:14 CEST Edwin Zimmerman wrote:
> I would have interest in it.
>
>
>
> --Edwin
>
>
>
>
>
> I'm no security expert, but we've got a big pile of serialization code that
> is kind of like JSON-p
On Sun, Jul 19, 2020, at 12:42, Stephen J. Turnbull wrote:
> > Sure - it'd have to be a new opcode at this point,
>
> Why? The REDUCE opcode invokes load_reduce which ... oh heck, just post it:
>
> > I just think the wrong decision was made in the first place,
>
> Which "first place", the earl
Random832 writes:
> I guess part of where I'm not sure I'm on solid ground is... is the
> pure-python version guaranteed to always exist and always be
> available under the name _Unpickler, or is that an implementation
> detail? I've been assuming that there was no such guarantee and any
> ch
Tragic! Pickle is relatively (?) fast and could be made more secure while
making any performance regression due to additional security optional.
Perhaps it is the objectives of pickle which are desirable:
- serialize/deserialize arbitrary objects
- binary representation
Or perhaps the docs cou
On Sat, Jul 18, 2020, at 12:54, Stephen J. Turnbull wrote:
> > I think I got all of them, but if you think there may be others
> > feel free to be an extra pair of eyes. But these overrides are not
> > available for the C version,
>
> That's going to be a sticking point, as many pickle use case
Random832 writes:
> On Thu, Jul 16, 2020, at 02:36, Stephen J. Turnbull wrote:
> > Random832 writes:
> >
> > > I was asking for the current Unpickler class, which currently has a
> > > whitelist hook for loading globals,
> >
> > Callables are globals in this sense.
>
> not all calla
On Thu, Jul 16, 2020, 2:30 AM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:
> [...]
>
> > How do I use JSON to serialise an arbitrary instance of some class?
>
> Ask Wes Turner about semantic JSON or whatever it is he frequently
> advocates for providing more type information
On Thu, Jul 16, 2020, at 02:36, Stephen J. Turnbull wrote:
> Random832 writes:
>
> > I was asking for the current Unpickler class, which currently has a
> > whitelist hook for loading globals,
>
> Callables are globals in this sense.
not all callables are globals, as has been pointed out attr
Random832 writes:
> I was asking for the current Unpickler class, which currently has a
> whitelist hook for loading globals,
Callables are globals in this sense. So overriding
Unpickler.find_class will allow you to restrict to specified
callables. It's not clear to me why you would want more
Steven D'Aprano writes:
> But if I'm distributing my code to others, the responsible thing to do
> is to think of the potential security risks about using pickle in my
> app, or library. What if they use it in ways that I didn't foresee, ways
> which *ought to be* safe except for my choice
On Wed, Jul 15, 2020, at 21:16, Chris Angelico wrote:
> Are you sure of that? I don't have any examples to hand, but are you
> able to pickle something identified as pkg.module.cls(x)?
This produces find_class('pkg.module', 'cls').
Doing pkg.module.cls.method produces find_class('builtins',
'
On Wed, Jul 15, 2020, at 07:40, Steven D'Aprano wrote:
> On Tue, Jul 14, 2020 at 09:47:15PM -0400, Random832 wrote:
>
> > I was asking for the current Unpickler class, which currently has a
> > whitelist hook for loading globals, to be modified to also have a
> > whitelist hook so that an applic
On Thu, Jul 16, 2020 at 11:13 AM Random832 wrote:
>
> On Wed, Jul 15, 2020, at 08:14, Chris Angelico wrote:
> > That's fair, but are you actually guaranteeing that it will never read
> > arbitrary attributes from objects?
>
> First of all, reading an attribute of an object in a pickle requires the
On Wed, Jul 15, 2020, at 08:14, Chris Angelico wrote:
> That's fair, but are you actually guaranteeing that it will never read
> arbitrary attributes from objects?
First of all, reading an attribute of an object in a pickle requires the
getattr function. Even currently, you can substitute your o
On Wed, Jul 15, 2020, at 07:54, Edwin Zimmerman wrote:
> The idea that the pickle module can be made "safe" is magical thinking.
> Pickle's attack surface is just too large and too powerful.
I don't think that makes something *inherently* unsafe, it just makes it
difficult to make it safe. The
On Wed, Jul 15, 2020 at 9:37 PM Steven D'Aprano wrote:
>
> On Wed, Jul 15, 2020 at 11:24:17AM +1000, Chris Angelico wrote:
> > So if you're distributing your code, then maybe you don't use pickle.
>
> Sure. What do I use to serialise my complex data structure? I guess I
> could write out the repr
Random832 [mailto:random...@fastmail.com] wrote:
> On Tue, Jul 14, 2020, at 21:24, Chris Angelico wrote:
> > I actively oppose it because it isn't possible. Anything that is safe
> > will not have all of pickle's functionality. A nerfed version of
> > pickle that can only unpickle a tiny handful of
On Tue, Jul 14, 2020 at 09:47:15PM -0400, Random832 wrote:
> I was asking for the current Unpickler class, which currently has a
> whitelist hook for loading globals, to be modified to also have a
> whitelist hook so that an application can provide a function that
> looks at a callable and its
On Wed, Jul 15, 2020 at 11:24:17AM +1000, Chris Angelico wrote:
> It's correct far more often than you might think. There's a LOT of
> code out there where the Python source code has the exact same
> external access permissions as its config files - often because
> there's no access to either.
Um
On Wed, 15 Jul 2020 09:45:06 +1000
Steven D'Aprano wrote:
>
> And that's the risk: can I guarantee that there is no clever scheme by
> which an attacker can fool me into unpickling malicious code? I need to
> be smarter than the attacker, and more imaginative, and to have thought
> as long and
On Tue, Jul 14, 2020, at 21:24, Chris Angelico wrote:
> I actively oppose it because it isn't possible. Anything that is safe
> will not have all of pickle's functionality. A nerfed version of
> pickle that can only unpickle a tiny handful of core data types is no
> better than other options that a
On Wed, Jul 15, 2020 at 11:00 AM Steven D'Aprano wrote:
>
> On Wed, Jul 15, 2020 at 09:55:03AM +1000, Chris Angelico wrote:
>
> > At that point, you are NOT running it with the "exact same access
> > permissions", are you? :)
>
> Indeed, and I did acknowledge that you were probably thinking about
On Wed, Jul 15, 2020 at 09:55:03AM +1000, Chris Angelico wrote:
> At that point, you are NOT running it with the "exact same access
> permissions", are you? :)
Indeed, and I did acknowledge that you were probably thinking about a
different scenario. But I was challenging your assertion that anyo
On Wed, Jul 15, 2020 at 9:46 AM Steven D'Aprano wrote:
>
> On Mon, Jul 13, 2020 at 09:56:45PM +1000, Chris Angelico wrote:
>
> > A pickle file (or equivalent blob in a database, or whatever) should
> > be considered equally as trusted as your source code. If you're
> > writing out a file that has
On Mon, Jul 13, 2020 at 09:56:45PM +1000, Chris Angelico wrote:
> A pickle file (or equivalent blob in a database, or whatever) should
> be considered equally as trusted as your source code. If you're
> writing out a file that has the exact same access permissions as your
> own source code, and th
On Mon, Jul 13, 2020 at 12:03 PM Edwin Zimmerman
wrote:
> I would have interest in it.
>
OK -- I'll see what I can do about pulling it out and putting it on gitHub.
Not sure I'll have the time to clean it up and make a nice package out of
it, but maybe there's some ideas in there worth sharing.
On Sat, Jul 11, 2020, at 20:15, Greg Ewing wrote:
> The set of callables that can be considered "safe" depends
> on the application, so there can't really be a generic
> "safe" option. If that were possible, it would no doubt
> already exist and be the default.
My main concern is wanting to make t
I would have interest in it.
--Edwin
I'm no security expert, but we've got a big pile of serialization code that is
kind of like JSON-pickly, but it will only deserialize known objects. it's a
bit of pain to declare what you want to work with, but it seems safer.
I also have a newe
I'm no security expert, but we've got a big pile of serialization code that
is kind of like JSON-pickly, but it will only deserialize known objects.
it's a bit of pain to declare what you want to work with, but it seems
safer.
I also have a newer system (built on top of dataclasses) that serialize
Looks like pyro4 (python remote objects) has moved to the serpent library
(as.literal_eval) [1]
> defaults to a safe serializer (serpent
https://pypi.python.org/pypi/serpent ) that supports many Python data types.
> supports different serializers (serpent, json, marshal, msgpack, pickle,
cloudpick
On Mon, Jul 13, 2020 at 8:58 PM Edwin Zimmerman wrote:
>
> On 7/11/2020 11:17 PM, Greg Ewing wrote:
>
> On 12/07/20 1:01 pm, Edwin Zimmerman wrote:
>
> As I see it, the unsafe callables (eval, exec, os.system, etc) are generally
> functions, and safe ones(int, list, dict) are generally classes, t
On 7/11/2020 11:17 PM, Greg Ewing wrote:
> On 12/07/20 1:01 pm, Edwin Zimmerman wrote:
>> As I see it, the unsafe callables (eval, exec, os.system, etc) are generally
>> functions, and safe ones(int, list, dict) are generally classes, though
>> there certainly would be exceptions.
>
> Where secur
On 12/07/20 1:01 pm, Edwin Zimmerman wrote:
As I see it, the unsafe callables (eval, exec, os.system, etc) are generally
functions, and safe ones(int, list, dict) are generally classes, though there
certainly would be exceptions.
Where security is concerned, "there certainly would be exceptio
The security problem arises from the fact that pickle will call arbitrary
functions and that it will unpickle arbitrary classes, not just the ones
that you might intend it to.
It seems to me that the way to make pickle safe is to limit what it can
call. Unpickle can take a list of classes and it w
If there were a configurable allow list of "safe" types, what in the stdlib
would and wouldn't be on the list?
On Sat, Jul 11, 2020, 9:16 PM Edwin Zimmerman
wrote:
> As I see it, the unsafe callables (eval, exec, os.system, etc) are
> generally functions, and safe ones(int, list, dict) are gener
As I see it, the unsafe callables (eval, exec, os.system, etc) are generally
functions, and safe ones(int, list, dict) are generally classes, though there
certainly would be exceptions.
Would it be too great of a breaking change to block function callables by
default? That might be an increment
On 12/07/20 8:54 am, Wes Turner wrote:
Would it be feasible to just NOP callables when safe=True?
This would break pickle, because calling constructors is
the way many objects are unpickled. And it's not easy to
tell which callables are safe to use as constructors and
which aren't.
--
Greg
___
On 12/07/20 5:31 am, Wes Turner wrote:
Is there already a way to load data and not code *with pickle*?
As far as I know, pickle has never been able to load code
objects.
The security problems come from the fact that by default a
pickle is able to *call* any module-level callable object
that it
The bottom line is that pickle should never be used in a security sensitive
context. Several years ago I spent about 5 minutes writing a custom pickle
fuzzer. It ran for about 60 seconds before segfaulting. Fortunately, the last
time I ran my fuzzer (about a year ago), all I could produce was
AFAIU, jsonpickle (and fill, cloud pickle,) will still execute arbitray
python (and ctypes) code.
Isn't pickle faster than C JSON?
Would it be feasible to just NOP callables when safe=True? Or would that be
pointless?
JSON5 is great but still doesn't handle e.g. complex fractions
On Sat, Jul 11
On Sat, Jul 11, 2020 at 4:24 PM Christopher Barker
wrote:
> NOTE: I've wanted for ages to make a "PYSON" format / module for when JSON
> is not quite enough. e.g. distinction between lists and tuples, dict keys
> that aren't strings
>
https://github.com/jsonpickle/jsonpickle
You're not the
On Sat, Jul 11, 2020 at 10:33 AM Wes Turner wrote:
> Is there already a way to load data and not code *with pickle*?
> https://docs.python.org/3/library/pickle.html
>
I'm not sure if this is what you mean, but there is:
ast.literal_eval()
which I *think* is safe.
NOTE: I've wanted for ages to
Would this accomplish something like:
pickle.load(safe=True) # or
pickle.safe_loads()
Is there already a way to load data and not code *with pickle*?
https://docs.python.org/3/library/pickle.html
On Sat, Jul 11, 2020, 11:01 AM Random832 wrote:
> The current practice, by overriding find_class,
43 matches
Mail list logo
