Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Wojciech S. Czarnecki]

Dnia 2021-02-03, o godz. 14:04:42
"'Axel Wagner' via golang-nuts"  napisał(a):

> Maybe it helps to point out that the statements "you should design your
> system to thoroughly validate input and reject it if it's invalid" and
> "there are contexts, where trying to be as flexible as possible in trying
> to make sense of an input" can both be true.

Of course that often both are desired. Just rarely we can attain both at once.
Thats why I emphasized "in this (security) context".

As for the OP problem: the best would be to have a flexible decoder that also
returns an indicator what unexpected format discrepancy it forgave. This would
be easy to do for static data; I see no easy solution for such signaling on 
streams
though.

TC,

-- 
Wojciech S. Czarnecki
 << ^oo^ >> OHIR-RIPE

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210203193627.23360769%40xmint.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

Maybe it helps to point out that the statements "you should design your
system to thoroughly validate input and reject it if it's invalid" and
"there are contexts, where trying to be as flexible as possible in trying
to make sense of an input" can both be true.

For example, I would agree that if you build a service, you should
generally try to not to be too liberal in what you accept. Because Hyrum's
Law  implies that over time, this would create
a more and more complex de-facto standard you (and others) need to
implement.
But I would also argue, that it's useful to have an interactive tool you
can just throw things to and have it try and make sense of it - if I just
find a Base64 string on the internet and want to know what it means, it's
inconvenient to have to fully specify or convert the format, especially if
I'm just doing what that tool would be doing anyway.
So, this argument is a false dichotomy. Both ends of the spectrum have
their place in practice.

I would also emphasize that we should give ways to the actual question. It
is frustrating to ask "how can I do X" and only being told "you shouldn't
want X". Of course we can present arguments for why we think X is not a
good idea, but answering the question should be first and foremost.
And we can always have a thorough discussion of whether such an API should
exist or not, once someone proposes to add it to the stdlib.

On Wed, Feb 3, 2021 at 1:37 PM Wojciech S. Czarnecki 
wrote:

> Dnia 2021-02-02, o godz. 22:26:10
> "hey...@gmail.com"  napisał(a):
>
> > > So having a “meta/relaxed decoder” usually leads to
> > specification/interoperability/security problems down the road
>
> > I respectfully disagree. Since it's only relaxed with regard to
> decoding,
> > it follows the robustness principle where you be liberal in what you
> accept.
>
> I disagree with such disagreement in this (security) context.
> "Robustness" stated as "accept lousy data" is against security principle
> "vet your input thorough".
>
> > Within a system, the encoding should be explicitly defined, but when
> that
> > system has to consume base64 data from outside, being liberal actually
> > avoids interoperability problems.
>
> In security context "avoids interoperability problems" may morph to more
> accurate "avoids preventing access to our systems by an adversary" - as
> adversaries are known to eagerly and clandestinely interoperate with our
> software using whatever means we left them to exploit. (Off the hat
> example: consuming "liberal" JSON input may allow an attacker to disrupt
> data guarded by a simple MAC scheme.)
>
> TC,
>
> --
> Wojciech S. Czarnecki
>  << ^oo^ >> OHIR-RIPE
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/20210203133700.36c529f9%40xmint
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfHT_dZqVC-bOkPenpiHz%2BtThgXb3s3p14tT%3DrKkb_nBvg%40mail.gmail.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Wojciech S. Czarnecki]

Dnia 2021-02-02, o godz. 22:26:10
"hey...@gmail.com"  napisał(a):

> > So having a “meta/relaxed decoder” usually leads to   
> specification/interoperability/security problems down the road

> I respectfully disagree. Since it's only relaxed with regard to decoding, 
> it follows the robustness principle where you be liberal in what you accept.

I disagree with such disagreement in this (security) context.
"Robustness" stated as "accept lousy data" is against security principle "vet 
your input thorough".

> Within a system, the encoding should be explicitly defined, but when that 
> system has to consume base64 data from outside, being liberal actually 
> avoids interoperability problems.

In security context "avoids interoperability problems" may morph to more 
accurate "avoids preventing access to our systems by an adversary" - as 
adversaries are known to eagerly and clandestinely interoperate with our 
software using whatever means we left them to exploit. (Off the hat example: 
consuming "liberal" JSON input may allow an attacker to disrupt data guarded by 
a simple MAC scheme.) 

TC,

-- 
Wojciech S. Czarnecki
 << ^oo^ >> OHIR-RIPE

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210203133700.36c529f9%40xmint.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[hey...@gmail.com]

> So having a “meta/relaxed decoder” usually leads to 
specification/interoperability/security problems down the road
I respectfully disagree. Since it's only relaxed with regard to decoding, 
it follows the robustness principle where you be liberal in what you accept.

Within a system, the encoding should be explicitly defined, but when that 
system has to consume base64 data from outside, being liberal actually 
avoids interoperability problems.
On Wednesday, February 3, 2021 at 1:10:32 PM UTC+8 ren...@ix.netcom.com 
wrote:

> Sorry it just doesn’t “feel right”. There are different encoding scheme as 
> laid out in the RFC. and other RFCs 
>  that cover 
> their uses.
>
> If you have a system that states “send us Base64 data” it is poorly 
> specified - better to state, send us Base64 data according to RFC 4648 
> base64url format or according to RFC-2045.
>
> In fact, the RFC states:
>
> "This encoding may be referred to as "base64url".  This encoding
>should not be regarded as the same as the "base64" encoding and
>should not be referred to as only "base64".  Unless clarified
>otherwise, "base64" refers to the base 64 in the previous section.”
>
> It also states:
>
> "If non-alphabet characters are ignored, instead of causing rejection
>of the entire encoding (as recommended), a covert channel that can be
>used to "leak" information is made possible."
>
> So having a “meta/relaxed decoder” usually leads to 
> specification/interoperability/security problems down the road. I realize 
> that in the “real world” you are often forced to interoperate with these 
> “bad” systems, but as most things in Go, better to be explicit and report 
> errors rather than be clever.
>
>
>
> On Feb 2, 2021, at 12:34 PM, 'Axel Wagner' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  wrote:
>
>> What “padding” are you referring to? Each must be 2 characters. And there 
>> is a standard that covers this https://tools.ietf.org/html/rfc4648
>>
>
> Yes, there indeed is. Section 5 describes a second encoding scheme, used 
> for URLs and the like. Section 3.2 also talks about the padding I'm 
> referring to (it's defined elsewhere in the standard) and mentions that, in 
> certain situations, it can be omitted. In particular, you can omit padding 
> and, in the decoder, implicitly pad to a multiple of 4 bytes.
>
> I don't really understand what's the argument is here. The question was if 
> it is possible to handle all four encoding schemas 
>  supported by the 
> Go base64 package in one swoop, because as-is, the API requires you to pick 
> one schema and just see if it returns an error. Roger provided, IMO, a 
> pretty good answer to that: You can wrap the io.Reader in one that 
> transparently rewrites any of the four into one well-known one, which can 
> then be handled by the corresponding decoder. His link provides the code 
> for an implementation of such a reader.
>  
>
>>
>> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>> 
>> Rogers approach seems like the best one to me - wrap the input in a 
>> custom `io.Reader` that transparently replaces `-_` with `+/` respectively 
>> (and drop trailing `=`). The bufio approach doesn't work, because there is 
>> no guarantee that one of the distinguishing characters is early in the 
>> stream and the "send it to multiple decoders" approach duplicates effort 
>> and wastes resources.
>>
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>>
>>> Reading through a bufio.Reader is often useful for these situations.
>>> You can peek the beginning of the input in order to decide which decoder 
>>> to use.
>>>
>>> Another option is to use the io.TeeReader to duplicate the reader,
>>> and then send one copy to each decoder.
>>> One will succeed, and give you the output.
>>> But you will need to drain the one that fails to prevent the TeeReader 
>>> form stalling.
>>>
>>>
>>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com 
>>> wrote:
>>>
 This question isn't about the decoded data, but about *which* base64 
 format is used - i.e. if it uses padding or not and what 2 characters are 
 used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's 
 easy 
 to tell which is used and just accept either (and padding can be viewed as 
 optional during decoding anyway).

 On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  
 wrote:

> Base64 is always ASCII. The encoded data may be in an arbitrary 
> format. You need to pass additional metadata or try and detect its 
> encoding. 
>
> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>
> 
> In case you find it helpful, here's a clone of the base64 command that 
> I wrote in Go. I did it precisely because I 

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[robert engels]

Sorry it just doesn’t “feel right”. There are different encoding scheme as laid 
out in the RFC. and other RFCs 
 that cover their 
uses.

If you have a system that states “send us Base64 data” it is poorly specified - 
better to state, send us Base64 data according to RFC 4648 base64url format or 
according to RFC-2045.

In fact, the RFC states:

"This encoding may be referred to as "base64url".  This encoding
   should not be regarded as the same as the "base64" encoding and
   should not be referred to as only "base64".  Unless clarified
   otherwise, "base64" refers to the base 64 in the previous section.”

It also states:

"If non-alphabet characters are ignored, instead of causing rejection
   of the entire encoding (as recommended), a covert channel that can be
   used to "leak" information is made possible."

So having a “meta/relaxed decoder” usually leads to 
specification/interoperability/security problems down the road. I realize that 
in the “real world” you are often forced to interoperate with these “bad” 
systems, but as most things in Go, better to be explicit and report errors 
rather than be clever.



> On Feb 2, 2021, at 12:34 PM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  > wrote:
> What “padding” are you referring to? Each must be 2 characters. And there is 
> a standard that covers this https://tools.ietf.org/html/rfc4648 
> 
> 
> Yes, there indeed is. Section 5 describes a second encoding scheme, used for 
> URLs and the like. Section 3.2 also talks about the padding I'm referring to 
> (it's defined elsewhere in the standard) and mentions that, in certain 
> situations, it can be omitted. In particular, you can omit padding and, in 
> the decoder, implicitly pad to a multiple of 4 bytes.
> 
> I don't really understand what's the argument is here. The question was if it 
> is possible to handle all four encoding schemas 
>  supported by the Go 
> base64 package in one swoop, because as-is, the API requires you to pick one 
> schema and just see if it returns an error. Roger provided, IMO, a pretty 
> good answer to that: You can wrap the io.Reader in one that transparently 
> rewrites any of the four into one well-known one, which can then be handled 
> by the corresponding decoder. His link provides the code for an 
> implementation of such a reader.
>  
> 
>> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts 
>> mailto:golang-nuts@googlegroups.com>> wrote:
>> 
>> 
>> Rogers approach seems like the best one to me - wrap the input in a custom 
>> `io.Reader` that transparently replaces `-_` with `+/` respectively (and 
>> drop trailing `=`). The bufio approach doesn't work, because there is no 
>> guarantee that one of the distinguishing characters is early in the stream 
>> and the "send it to multiple decoders" approach duplicates effort and wastes 
>> resources.
>> 
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon > > wrote:
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder to 
>> use.
>> 
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader form 
>> stalling.
>> 
>> 
>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com 
>>  wrote:
>> This question isn't about the decoded data, but about *which* base64 format 
>> is used - i.e. if it uses padding or not and what 2 characters are used 
>> outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to 
>> tell which is used and just accept either (and padding can be viewed as 
>> optional during decoding anyway).
>> 
>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels > wrote:
>> Base64 is always ASCII. The encoded data may be in an arbitrary format. You 
>> need to pass additional metadata or try and detect its encoding. 
>> 
>>> On Feb 2, 2021, at 6:50 AM, roger peppe > wrote:
>>> 
>>> 
>>> In case you find it helpful, here's a clone of the base64 command that I 
>>> wrote in Go. I did it precisely because I wanted to be able to decode any 
>>> encoding scheme interchangeably.
>>> 
>>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go 
>>> 
>>> 
>>> I agree that it might be useful to have some of this functionality 
>>> available in the standard library.
>>> 
>>>   cheers,
>>> rog.
>>> 
>>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com <> > 
>>> wrote:
>>> Hi,
>>> 
>>> I have an io.Reader whose content is encoded in base64 with encoding type 
>>> unknown. 

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[hey...@gmail.com]

Your translate reader works really well, thanks for sharing it.

I have seen code that tried to decode base64 four times in the wild, which 
led me to posting this, hope something like this could be incorporated into 
the standard library. 

On Tuesday, February 2, 2021 at 8:50:08 PM UTC+8 rog wrote:

> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
>
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>
> I agree that it might be useful to have some of this functionality 
> available in the standard library.
>
>   cheers,
> rog.
>
> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>
>> Hi,
>>
>> I have an io.Reader whose content is encoded in base64 with encoding type 
>> unknown. Since there shouldn't be any ambiguity between the two, is it 
>> possible to make the base64 automatically pick the right one to decode?
>>
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>>
>> Is there a solution to this problem?
>>
>> Thanks in advance.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/15c0911e-cf08-48e2-b979-054ee7525d97n%40googlegroups.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  wrote:

> What “padding” are you referring to? Each must be 2 characters. And there
> is a standard that covers this https://tools.ietf.org/html/rfc4648
>

Yes, there indeed is. Section 5 describes a second encoding scheme, used
for URLs and the like. Section 3.2 also talks about the padding I'm
referring to (it's defined elsewhere in the standard) and mentions that, in
certain situations, it can be omitted. In particular, you can omit padding
and, in the decoder, implicitly pad to a multiple of 4 bytes.

I don't really understand what's the argument is here. The question was if
it is possible to handle all four encoding schemas
 supported by the Go
base64 package in one swoop, because as-is, the API requires you to pick
one schema and just see if it returns an error. Roger provided, IMO, a
pretty good answer to that: You can wrap the io.Reader in one that
transparently rewrites any of the four into one well-known one, which can
then be handled by the corresponding decoder. His link provides the code
for an implementation of such a reader.


>
> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
> 
> Rogers approach seems like the best one to me - wrap the input in a custom
> `io.Reader` that transparently replaces `-_` with `+/` respectively (and
> drop trailing `=`). The bufio approach doesn't work, because there is no
> guarantee that one of the distinguishing characters is early in the stream
> and the "send it to multiple decoders" approach duplicates effort and
> wastes resources.
>
> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder
>> to use.
>>
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader
>> form stalling.
>>
>>
>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com
>> wrote:
>>
>>> This question isn't about the decoded data, but about *which* base64
>>> format is used - i.e. if it uses padding or not and what 2 characters are
>>> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy
>>> to tell which is used and just accept either (and padding can be viewed as
>>> optional during decoding anyway).
>>>
>>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels 
>>> wrote:
>>>
 Base64 is always ASCII. The encoded data may be in an arbitrary format.
 You need to pass additional metadata or try and detect its encoding.

 On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:

 
 In case you find it helpful, here's a clone of the base64 command that
 I wrote in Go. I did it precisely because I wanted to be able to decode any
 encoding scheme interchangeably.

 https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go

 I agree that it might be useful to have some of this functionality
 available in the standard library.

   cheers,
 rog.

 On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:

> Hi,
>
> I have an io.Reader whose content is encoded in base64 with encoding
> type unknown. Since there shouldn't be any ambiguity between the two, is 
> it
> possible to make the base64 automatically pick the right one to decode?
>
> Currently I have to read everything out to pin down the encoding,
> which defeats the purpose of using an io.Reader.
>
> Is there a solution to this problem?
>
> Thanks in advance.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
> 
> .
>
 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
 
 .

 --
 You received this 

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Robert Engels]

What “padding” are you referring to? Each must be 2 characters. And there is a 
standard that covers this https://tools.ietf.org/html/rfc4648

> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> Rogers approach seems like the best one to me - wrap the input in a custom 
> `io.Reader` that transparently replaces `-_` with `+/` respectively (and drop 
> trailing `=`). The bufio approach doesn't work, because there is no guarantee 
> that one of the distinguishing characters is early in the stream and the 
> "send it to multiple decoders" approach duplicates effort and wastes 
> resources.
> 
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder to 
>> use.
>> 
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader form 
>> stalling.
>> 
>> 
>>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com wrote:
>>> This question isn't about the decoded data, but about *which* base64 format 
>>> is used - i.e. if it uses padding or not and what 2 characters are used 
>>> outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to 
>>> tell which is used and just accept either (and padding can be viewed as 
>>> optional during decoding anyway).
>>> 
 On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:
 Base64 is always ASCII. The encoded data may be in an arbitrary format. 
 You need to pass additional metadata or try and detect its encoding. 
 
>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>> 
> 
> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
> 
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
> 
> I agree that it might be useful to have some of this functionality 
> available in the standard library.
> 
>   cheers,
> rog.
> 
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>> Hi,
>> 
>> I have an io.Reader whose content is encoded in base64 with encoding 
>> type unknown. Since there shouldn't be any ambiguity between the two, is 
>> it possible to make the base64 automatically pick the right one to 
>> decode?
>> 
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>> 
>> Is there a solution to this problem?
>> 
>> Thanks in advance.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to golang-nuts...@googlegroups.com.
>>> 
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGGkOX9HzF6e178Cu6BD9Hg4x5LT7AutHkUDPvyK%2BFmYg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe 

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

Rogers approach seems like the best one to me - wrap the input in a custom
`io.Reader` that transparently replaces `-_` with `+/` respectively (and
drop trailing `=`). The bufio approach doesn't work, because there is no
guarantee that one of the distinguishing characters is early in the stream
and the "send it to multiple decoders" approach duplicates effort and
wastes resources.

On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:

> Reading through a bufio.Reader is often useful for these situations.
> You can peek the beginning of the input in order to decide which decoder
> to use.
>
> Another option is to use the io.TeeReader to duplicate the reader,
> and then send one copy to each decoder.
> One will succeed, and give you the output.
> But you will need to drain the one that fails to prevent the TeeReader
> form stalling.
>
>
> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com
> wrote:
>
>> This question isn't about the decoded data, but about *which* base64
>> format is used - i.e. if it uses padding or not and what 2 characters are
>> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy
>> to tell which is used and just accept either (and padding can be viewed as
>> optional during decoding anyway).
>>
>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels 
>> wrote:
>>
>>> Base64 is always ASCII. The encoded data may be in an arbitrary format.
>>> You need to pass additional metadata or try and detect its encoding.
>>>
>>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>>>
>>> 
>>> In case you find it helpful, here's a clone of the base64 command that I
>>> wrote in Go. I did it precisely because I wanted to be able to decode any
>>> encoding scheme interchangeably.
>>>
>>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>>>
>>> I agree that it might be useful to have some of this functionality
>>> available in the standard library.
>>>
>>>   cheers,
>>> rog.
>>>
>>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>>>
 Hi,

 I have an io.Reader whose content is encoded in base64 with encoding
 type unknown. Since there shouldn't be any ambiguity between the two, is it
 possible to make the base64 automatically pick the right one to decode?

 Currently I have to read everything out to pin down the encoding, which
 defeats the purpose of using an io.Reader.

 Is there a solution to this problem?

 Thanks in advance.


 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Amnon]

Reading through a bufio.Reader is often useful for these situations.
You can peek the beginning of the input in order to decide which decoder to 
use.

Another option is to use the io.TeeReader to duplicate the reader,
and then send one copy to each decoder.
One will succeed, and give you the output.
But you will need to drain the one that fails to prevent the TeeReader form 
stalling.


On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com wrote:

> This question isn't about the decoded data, but about *which* base64 
> format is used - i.e. if it uses padding or not and what 2 characters are 
> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy 
> to tell which is used and just accept either (and padding can be viewed as 
> optional during decoding anyway).
>
> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:
>
>> Base64 is always ASCII. The encoded data may be in an arbitrary format. 
>> You need to pass additional metadata or try and detect its encoding. 
>>
>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>>
>> 
>> In case you find it helpful, here's a clone of the base64 command that I 
>> wrote in Go. I did it precisely because I wanted to be able to decode any 
>> encoding scheme interchangeably.
>>
>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>>
>> I agree that it might be useful to have some of this functionality 
>> available in the standard library.
>>
>>   cheers,
>> rog.
>>
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>>
>>> Hi,
>>>
>>> I have an io.Reader whose content is encoded in base64 with encoding 
>>> type unknown. Since there shouldn't be any ambiguity between the two, is it 
>>> possible to make the base64 automatically pick the right one to decode?
>>>
>>> Currently I have to read everything out to pin down the encoding, which 
>>> defeats the purpose of using an io.Reader.
>>>
>>> Is there a solution to this problem?
>>>
>>> Thanks in advance.
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
>>  
>> 
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

This question isn't about the decoded data, but about *which* base64 format
is used - i.e. if it uses padding or not and what 2 characters are used
outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to
tell which is used and just accept either (and padding can be viewed as
optional during decoding anyway).

On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:

> Base64 is always ASCII. The encoded data may be in an arbitrary format.
> You need to pass additional metadata or try and detect its encoding.
>
> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>
> 
> In case you find it helpful, here's a clone of the base64 command that I
> wrote in Go. I did it precisely because I wanted to be able to decode any
> encoding scheme interchangeably.
>
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>
> I agree that it might be useful to have some of this functionality
> available in the standard library.
>
>   cheers,
> rog.
>
> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>
>> Hi,
>>
>> I have an io.Reader whose content is encoded in base64 with encoding type
>> unknown. Since there shouldn't be any ambiguity between the two, is it
>> possible to make the base64 automatically pick the right one to decode?
>>
>> Currently I have to read everything out to pin down the encoding, which
>> defeats the purpose of using an io.Reader.
>>
>> Is there a solution to this problem?
>>
>> Thanks in advance.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGFqX_aBoecB-BAMeFaDVQJnMh%2BbxTMig4wyBuM5RLWOw%40mail.gmail.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Robert Engels]

Base64 is always ASCII. The encoded data may be in an arbitrary format. You 
need to pass additional metadata or try and detect its encoding. 

> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
> 
> 
> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
> 
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
> 
> I agree that it might be useful to have some of this functionality available 
> in the standard library.
> 
>   cheers,
> rog.
> 
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>> Hi,
>> 
>> I have an io.Reader whose content is encoded in base64 with encoding type 
>> unknown. Since there shouldn't be any ambiguity between the two, is it 
>> possible to make the base64 automatically pick the right one to decode?
>> 
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>> 
>> Is there a solution to this problem?
>> 
>> Thanks in advance.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[roger peppe]

In case you find it helpful, here's a clone of the base64 command that I
wrote in Go. I did it precisely because I wanted to be able to decode any
encoding scheme interchangeably.

https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go

I agree that it might be useful to have some of this functionality
available in the standard library.

  cheers,
rog.

On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:

> Hi,
>
> I have an io.Reader whose content is encoded in base64 with encoding type
> unknown. Since there shouldn't be any ambiguity between the two, is it
> possible to make the base64 automatically pick the right one to decode?
>
> Currently I have to read everything out to pin down the encoding, which
> defeats the purpose of using an io.Reader.
>
> Is there a solution to this problem?
>
> Thanks in advance.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.