Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Just re-ran this (2 years later) with:
  go version go1.21.1 linux/amd64
  Ubuntu 22.04.3 LTS

and there is a new limit that is unhappy:

  too much data, last section SXCOFFTOC (2097170064, over 2e+09 bytes)
  too much data, last section SDATA (2097170064, over 2e+09 bytes)
*  too much data, last section SCOVERAGE_COUNTER (2097373648, over 2e+09 
bytes)*

(Which makes sense).

https://pkg.go.dev/runtime/coverage
https://github.com/golang/go/blob/master/src/cmd/link/internal/ld/data.go

Glen


On Monday, September 20, 2021 at 8:29:30 PM UTC-4 Glen Newton wrote:

> Hello,
>
> Thanks for this, and also sorry for the lazy question.
>
> My testing on my machine indicates a limit of 2GB total limit on compile: 
>
> too much data in section SXCOFFTOC (over 2e+09 bytes)
> too much data in section SDATA (over 2e+09 bytes)
>
> I have built a bash script that creates large random files of a fixed size 
> and then generates a short Go program that embeds them, increasing the 
> number of files with each run, so you can see when your compile fails. 
> You can find the bash and instructions here: 
> https://github.com/gnewton/test_go_embed
>
> I just threw it together, so might be a little rough around the edges. 
> Feedback welcome.
>
> Question: Wondering why the above error complains about too much data in 
> *two* sections: are the embedded files stored across multiple sections (I 
> know nothing of how this is done internally to the elf format)?
>
> Thanks,
> Glen
>
> On Monday, September 20, 2021 at 5:11:45 PM UTC-4 Ian Lance Taylor wrote:
>
>> On Mon, Sep 20, 2021 at 2:04 PM Glen Newton  wrote: 
>> > 
>> > I am testing this with an increasing number of 1GB files. At 3 files, 
>> and I am getting this error: 
>> > 
>> > compile: writing output: write $WORK/b001/_pkg_.a: no space left on 
>> device 
>> > 
>> > I haven't been able to find how to change $WORK to point to a larger 
>> partition. Suggestions? 
>>
>> Set the TMPDIR environment variable to control temporary files in 
>> general, or the GOTMPDIR environment variable to control just 
>> temporary files created by the go tool. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/28cad6f3-7f37-4b65-9004-00e5c8229f41n%40googlegroups.com.

[go-nuts] Re: are Go generics more like C++ or more like Java?

[Glen Newton]

>- exotic use-cases are not supported (for example having an integer 
*constant* as *generic parameter*, such as you see in C++ fixed size matrix 
templates, is not supported)

Does a longer (but not necessarily exhaustive) list of the unsupported 
exotic use cases exist? This would be useful to understand all of the 
situations where Go generics cannot (or should not?) be used.  

Glen

On Friday, February 25, 2022 at 5:13:01 AM UTC-5 markus@gain.pro wrote:

> I think the consensus in the Go community is to refrain from comparing Go 
> language features with other programming languages. Rationale ~:
>
> - it is highly contentious
> - it is very difficult to answer, it's like asking "is purple more blue or 
> more red"
> - no matter the answer, it will not help you a lot to write better 
> programs or be more productive in Go
>
> However, maybe your real question is more like "what are/were the guiding 
> principals for generics in Go?".   As far as I understand, generics in Go 
> follow the same principles as other language features in Go:
>
> - compilation should be very fast, but trade-offs are allowed so it does 
> not need to be maximally fast
> - language features should be as orthogonal as possible, but trade-offs 
> are allowed so they don't need to be maximally orthogonal
> - it should be a simple as possible but no simpler
> - ... 
>
> For generics this has resulted in:
>
> - exotic use-cases are not supported (for example having an integer 
> *constant* as *generic parameter*, such as you see in C++ fixed size matrix 
> templates, is not supported)
> - it must be possible to express that T is comparable (f.e. min(a, b T) T 
> must be possible to write)
> - it must be possible restrict T to a concrete list of types 
> - it must be possible to restrict T to a type with method set that equals 
> that of some interface (I *personally* think this violates the principle of 
> orthogonality too much and will lead to confusion and design debate)  
>
> Of course, neither list is exhaustive and is only my personal 
> understanding of things.
>
> -Markus
>
> On Wednesday, February 23, 2022 at 9:41:14 PM UTC+1 Jason E. Aten wrote:
>
>> Back in 2009, Russ wrote a blog on generics, talking about the tradeoffs 
>> in providing generics:
>>
>> "The generic dilemma is this: *do you want slow programmers, slow 
>> compilers and bloated binaries, or slow execution times? " -- *
>> https://research.swtch.com/generic
>>
>> I haven't had the bandwidth to follow the generics detailed discussions.  
>> I was just curious, at a high level, for creating a mental model of Go's 
>> generics: which approach was taken?
>>
>> Thanks!
>>
>> J
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/25a18a9a-45ae-43f7-b403-4cdf7f310b07n%40googlegroups.com.

Re: [go-nuts] Float32 -> string --> Float32 round trip not possible for strconv.FormatFloat 'b' ?

[Glen Newton]

Thanks to both of you for this! Appreciated.  :-)

Glen

On Thursday, December 9, 2021 at 11:20:11 PM UTC-5 kortschak wrote:

> On Thu, 2021-12-09 at 17:22 -0800, Kurtis Rader wrote:
> > Note that the documentation for ParseFloat states: "ParseFloat
> > accepts decimal and hexadecimal floating-point number syntax." In
> > other words, it explicitly does not support the binary format emitted
> > by FormatFloat('b'). I don't know why it doesn't but I'd bet that a
> > review of old issues and changes to the source would probably shed
> > some light on the reason.
> >
>
> This gives some background: https://github.com/golang/go/issues/12518
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2c3e2a8a-df77-42bc-92fb-6bbdf4e2263fn%40googlegroups.com.

[go-nuts] Float32 -> string --> Float32 round trip not possible for strconv.FormatFloat 'b' ?

[Glen Newton]

Hello,

I am using strconv.FormatFloat[1] and strconv.ParseFloat[2]to convert 
float32s back-and-forth in a library I am working on. In my tests I decided 
to test all formats for FormatFloat ('b', 'e', 'E', 'f', 'g', 'G', 'x', 
'X'). However, it seems like the 'b' format is not supported by ParseFloat 
(see [3]).

Is this correct? Is my code showing this correct?[3]
If Yes, is there some reason why the one format ('b') is not supported by 
ParseFloat?

My question is more one driven by curiosity rather than a need for this 
format. It was a surprise to me that one of the formats generated by the 
complement function in the same module was not supported.

Thanks,
Glen

[1]https://pkg.go.dev/strconv#FormatFloat
[2]https://pkg.go.dev/strconv#ParseFloat
[3]https://go.dev/play/p/b_GbyG_CFo6

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cd22042a-cd35-484b-af86-9f3d6f918376n%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Oh, regarding the comments about breaking tool chains: valgrind fails on 
this binary:

$ valgrind --version
valgrind-3.16.1
$ valgrind ./goembeddb -S -q
valgrind: mmap(0x54e000, 1977884672) failed in UME with error 22 (Invalid 
argument).
valgrind: this can be caused by executables with very large text, data or 
bss segments.
$ 


Previously reported: https://bugs.kde.org/show_bug.cgi?id=290061

Glen

On Thursday, September 23, 2021 at 8:15:36 PM UTC-4 Glen Newton wrote:

> I have a MWE here: https://github.com/gnewton/goembeddb
>
> TLDR: a MWE embedding a key-value store file of 20 million k/v pairs 
> (1.9GB) into a Go binary and using the db from the binary.
>
> The same Go program that writes the db then is recompiled, embedding the 
> db in the newly compiled Go binary. Size: ~1.9GB
>
> Then the same Go program is used to read the embedded db.
>
> *To read 1 random record from the embedded 1.9GB 20 million record 
> key-value db, takes 133.166µs, 2244k (~2.2MB) resident. Cold start.*
> This does not seem to bad, for this particular use case.  :-)
>
> I'd be interested in trying out BBolt <https://github.com/etcd-io/bbolt> 
> or Badger <https://github.com/dgraph-io/badger>, as the K/V db I am using 
> here (cdb https://en.wikipedia.org/wiki/Cdb_(software) ) does not support 
> prefix or range scans or buckets.
>
> Comments welcome.
>
> Thanks,
> Glen
>
> On Wednesday, September 22, 2021 at 1:22:53 PM UTC-4 Howard C. Shaw III 
> wrote:
>
>> Before the addition of binary packaged assets into Go as a standard 
>> library feature, there were various tools to accomplish the same task. Some 
>> of them, such as https://github.com/GeertJohan/go.rice , could use an 
>> alternate embedding. Basically, instead of having the binary files packaged 
>> as Go code in the executable itself, they simply appended a .zip file to 
>> the binary, and accessed that directly. 
>>
>> Now, I am not suggesting that as a direct solution (though it may be, as 
>> Go does apparently support Zip64 which does away with the 4GB limit), but 
>> perhaps instead looking at how it does what it does, and adapting that to 
>> simply appending a sqlite or other single-file database format to the Go 
>> binary and using the same basic technique for accessing it.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/4df6d635-630d-42ae-8106-a4dbc426ed17n%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

I have a MWE here: https://github.com/gnewton/goembeddb

TLDR: a MWE embedding a key-value store file of 20 million k/v pairs 
(1.9GB) into a Go binary and using the db from the binary.

The same Go program that writes the db then is recompiled, embedding the db 
in the newly compiled Go binary. Size: ~1.9GB

Then the same Go program is used to read the embedded db.

*To read 1 random record from the embedded 1.9GB 20 million record 
key-value db, takes 133.166µs, 2244k (~2.2MB) resident. Cold start.*
This does not seem to bad, for this particular use case.  :-)

I'd be interested in trying out BBolt  or 
Badger , as the K/V db I am using here 
(cdb https://en.wikipedia.org/wiki/Cdb_(software) ) does not support prefix 
or range scans or buckets.

Comments welcome.

Thanks,
Glen

On Wednesday, September 22, 2021 at 1:22:53 PM UTC-4 Howard C. Shaw III 
wrote:

> Before the addition of binary packaged assets into Go as a standard 
> library feature, there were various tools to accomplish the same task. Some 
> of them, such as https://github.com/GeertJohan/go.rice , could use an 
> alternate embedding. Basically, instead of having the binary files packaged 
> as Go code in the executable itself, they simply appended a .zip file to 
> the binary, and accessed that directly. 
>
> Now, I am not suggesting that as a direct solution (though it may be, as 
> Go does apparently support Zip64 which does away with the 4GB limit), but 
> perhaps instead looking at how it does what it does, and adapting that to 
> simply appending a sqlite or other single-file database format to the Go 
> binary and using the same basic technique for accessing it.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/71d65ad9-080c-453f-918b-086a9f56566fn%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Hello Ian,

Yes, I think it is time I explain the 'why' of my inquiries into this. :-)

My use case is this: Go, with its fast startup, pretty fast execution and 
pretty small memory footprint, as well as it's ability to deploy as a 
single binary, makes it a great language for cloud function-as-a-service 
(FAAS). 

My specific FAAS use case is for static, read-only databases: I am looking 
at embedding modest sized (few GB to 10s of GB) read-only databases in the 
Go binary. 

This makes it possible to avoid the cost/complication of either using a 
cloud db, or storing the db in (in the case of AWS) on a file system like 
EFS (NFS), which the lambda has to mount. The databases are only updated 
every couple of months / yearly.
This is perhaps rather an obscure use case, but one that I think a number 
of people would want to take advantage of.

>Note that such files are going to take a long time to create and will be 
unwieldy to use.
Agreed. On my older desktop (Dell 7010 Kubuntu 20.10 4core i5 16GM) it 
takes 45s to compile a trivial Go program that embeds three 500GB files.

Actually, this is no different from those who want the simplicity and cost 
savings of serving an entire static web site from embedded files. The 
primary difference is the scale.

Thanks,
Glen


On Tuesday, September 21, 2021 at 1:28:12 PM UTC-4 Ian Lance Taylor wrote:

> On Tue, Sep 21, 2021 at 6:23 AM Glen Newton  wrote:
> >
> > Looking at https://groups.google.com/g/golang-codereviews/c/xkHLQHidF5s 
> and https://github.com/golang/go/issues/9862 and the code to which they 
> refer, it seems to me that the limit is 2GB independent of platform (is 
> this true?), as the linker is limited to 2e9.
> > Again, should have done more of my homework! :-)
> >
> > 
> https://github.com/golang/go/blob/master/src/cmd/internal/obj/objfile.go#L305
> >
> > // cutoff is the maximum data section size permitted by the linker
> > // (see issue #9862). const cutoff = 2e9 // 2 GB (or so; looks better in 
> errors than 2^31)
> > const cutoff = 2e9 // 2 GB (or so; looks better in errors than 2^31)
> >
> > The comment indicates that this is a limit in the linker; is this a 
> limit in the elf format? If No, could the linker et al be modified to 
> accept larger static data?
>
> You're right: it does look like cmd/link imposes a 2G total limit.
> This is not a limitation of the 64-bit ELF format. It should be
> possible to make it larger.
>
> Note that such files are going to take a long time to create and will
> be unwieldy to use. While it should be possible to increase the size,
> I would not be surprised if you hit other limits fairly quickly.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9aee61f1-be26-45a8-96f6-b06ff65983ban%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Hello,

Looking at https://groups.google.com/g/golang-codereviews/c/xkHLQHidF5s and 
https://github.com/golang/go/issues/9862 and the code to which they refer, 
it seems to me that the limit is 2GB independent of platform (is this 
true?), as the linker is limited to 2e9.
Again, should have done more of my homework!  :-)

https://github.com/golang/go/blob/master/src/cmd/internal/obj/objfile.go#L305

// cutoff is the maximum data section size permitted by the linker 
// (see issue #9862). const cutoff = 2e9 // 2 GB (or so; looks better in 
errors than 2^31)
const cutoff = 2e9 // 2 GB (or so; looks better in errors than 2^31)

The comment indicates that this is a limit in the linker; is this a limit 
in the elf format? If No, could the linker et al be modified to accept 
larger static data?

Thanks,
Glen

On Monday, September 20, 2021 at 8:29:30 PM UTC-4 Glen Newton wrote:

> Hello,
>
> Thanks for this, and also sorry for the lazy question.
>
> My testing on my machine indicates a limit of 2GB total limit on compile: 
>
> too much data in section SXCOFFTOC (over 2e+09 bytes)
> too much data in section SDATA (over 2e+09 bytes)
>
> I have built a bash script that creates large random files of a fixed size 
> and then generates a short Go program that embeds them, increasing the 
> number of files with each run, so you can see when your compile fails. 
> You can find the bash and instructions here: 
> https://github.com/gnewton/test_go_embed
>
> I just threw it together, so might be a little rough around the edges. 
> Feedback welcome.
>
> Question: Wondering why the above error complains about too much data in 
> *two* sections: are the embedded files stored across multiple sections (I 
> know nothing of how this is done internally to the elf format)?
>
> Thanks,
> Glen
>
> On Monday, September 20, 2021 at 5:11:45 PM UTC-4 Ian Lance Taylor wrote:
>
>> On Mon, Sep 20, 2021 at 2:04 PM Glen Newton  wrote: 
>> > 
>> > I am testing this with an increasing number of 1GB files. At 3 files, 
>> and I am getting this error: 
>> > 
>> > compile: writing output: write $WORK/b001/_pkg_.a: no space left on 
>> device 
>> > 
>> > I haven't been able to find how to change $WORK to point to a larger 
>> partition. Suggestions? 
>>
>> Set the TMPDIR environment variable to control temporary files in 
>> general, or the GOTMPDIR environment variable to control just 
>> temporary files created by the go tool. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/85284731-937b-4253-a6b5-f345d7f5ccc4n%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Hello,

Thanks for this, and also sorry for the lazy question.

My testing on my machine indicates a limit of 2GB total limit on compile: 

too much data in section SXCOFFTOC (over 2e+09 bytes)
too much data in section SDATA (over 2e+09 bytes)

I have built a bash script that creates large random files of a fixed size 
and then generates a short Go program that embeds them, increasing the 
number of files with each run, so you can see when your compile fails. 
You can find the bash and instructions here: 
https://github.com/gnewton/test_go_embed

I just threw it together, so might be a little rough around the edges. 
Feedback welcome.

Question: Wondering why the above error complains about too much data in 
*two* sections: are the embedded files stored across multiple sections (I 
know nothing of how this is done internally to the elf format)?

Thanks,
Glen

On Monday, September 20, 2021 at 5:11:45 PM UTC-4 Ian Lance Taylor wrote:

> On Mon, Sep 20, 2021 at 2:04 PM Glen Newton  wrote:
> >
> > I am testing this with an increasing number of 1GB files. At 3 files, 
> and I am getting this error:
> >
> > compile: writing output: write $WORK/b001/_pkg_.a: no space left on 
> device
> >
> > I haven't been able to find how to change $WORK to point to a larger 
> partition. Suggestions?
>
> Set the TMPDIR environment variable to control temporary files in
> general, or the GOTMPDIR environment variable to control just
> temporary files created by the go tool.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/65e20ea9-cbe4-4cf1-823d-a8fa063ae44fn%40googlegroups.com.

Re: [go-nuts] What is the total max size of embed content?

[Glen Newton]

Thanks.

I am testing this with an increasing number of 1GB files. At 3 files, and I 
am getting this error:

compile: writing output: write $WORK/b001/_pkg_.a: no space left on device

I haven't been able to find how to change $WORK to point to a larger 
partition. Suggestions?

Oh, running:   
  go1.17.1.linux-amd64
  go version go1.17.1 linux/amd64
  Linux OptiPlex-7010 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 
UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


Thanks,
Glen


On Monday, September 20, 2021 at 12:25:08 PM UTC-4 Ian Lance Taylor wrote:

> On Mon, Sep 20, 2021 at 9:20 AM Glen Newton  wrote:
> >
> > The maximum size for any single embedded file as []byte is 4GB. What is 
> the total maximum size for all embedded files included this way in a Go 
> binary?
> > Also, are there any platform dependencies?
>
> The total maximum size for all embedded files depends on what the
> platform is able to support. And, yes, there are platform
> dependencies. Most 32-bit platforms can't support a total size larger
> than 2G. Most 64-bit platforms have much larger limits, though I
> don't know specifically what they are offhand.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2c69cff2-fd89-449c-8afb-c98236a09543n%40googlegroups.com.

[go-nuts] What is the total max size of embed content?

[Glen Newton]

Hello,

The maximum size for any single embedded file as []byte is 4GB. What is the 
total maximum size for all embedded files included this way in a Go binary?
Also, are there any platform dependencies?

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ab0e928c-fb52-44ff-b242-718c99da819fn%40googlegroups.com.

[go-nuts] Supporting contained zip archives inside of tar and zip archives

[Glen Newton]

zip archives contain files and directories. Often, some of the files 
contained in zip archives are also archives, zip and tar(.gz|bz2). 
Sometimes, these contained archives also contain such archives (and so 
on...).

Reading a zip file requires an io.ReaderAt in the func zip.NewReader(r 
io.ReaderAt, size int64) (*Reader, error) This function returns a 
*zip.Reader, which exposes an array of *zip.File, which can be opened with 
func (f *File) Open() (io.ReadCloser, error).

So reading any zip archives contained in a zip archive is not supported, as 
the  zip.Reader[n].File.Open() returns a ReadCloser, not an io.ReaderAt.

Others have encountered this issue, with some suggested solutions: 
https://stackoverflow.com/questions/40245442/handling-nested-zip-files-with-archive-zip

Reading tar(.gz|bz2) archives that are found in a zip archive are not 
problematic, as archive/tar.NewReader, compress/gzip.NewReader and 
compress/bzip2.NewReader all require io.Reader and return io.Reader. This 
is also true for tar archives that are contained in tar archives.

tar archives that contain zip files are not supported, as archive/tar does 
not expose an io.ReaderAt. Often (usually?), on-disk tar archives are 
gzipped/bzip2ed, and compress/gzip and compress/bzip2 also do not expose an 
io.ReaderAt that could be passed to archive/tar.

Let's take a step back: usually, a reader will either correspond to an 
ephemeral stream that comes in through a pipe or a socket, or it 
corresponds to an on-disk static file. An io.ReaderAt for the former is 
problematic (needing to store maybe the whole stream to allow 
rewind,seek,etc...); Here I am looking to address the use case of on-disk 
archives
supporting exposing a ReaderAt. In this case, any depth of zip/tar 
combinations could be supported through a recursive hierarchy of 
io.ReaderAts, with rewinds (probably) bubbling up to the top lever 
ioReaderAt, which could support io.ReaderAt.ReadAt, as it has the on-disk 
bytes to work with.

Proposal:

 Add to archive/zip:

type ReaderAt struct {
File[]*File
Comment string
// contains filtered or unexported fields
}

// This would be used at the 0th level and opens an on-disk zip archive
func OpenReaderAt(name string) (*zip.ReaderAt, error)

// This would return an error if the *zip.File was not contained in a
// zip.ReaderAt
func (f *File) OpenAt() (io.ReaderAt, error)

 Add archive/tar:
type ReaderAt struct {
// contains filtered or unexported fields
}

func OpenReaderAt(name string) (*tar.ReaderAt, error)
func NewReaderAt(r io.ReaderAt) (*tar.ReaderAt, error)
func (ra *ReaderAt) ReadAt(p []byte, off int64) (n int, err error)

 Add compress/gzip
func OpenReaderAt(f *io.File) (io.ReaderAt, error)

 Add compress/bzip2
func OpenReaderAt(f *io.File) (io.ReaderAt, error)


// zip containing zip example

// foo.zip contains: a.txt, bar.zip
//   bar.zip contains: x.txt
//
fooZipReaderAt := zip.OpenReaderAt("foo.zip")
for _, fFoo := range fooZipReaderAt.File {
if strings.HasSuffix(fFoo.Name, ".zip") {
   // ReaderAt for bar.zip
   barReaderAt, err := fFoo.OpenAt()

   // Open zip for bar.zip
   barZipReaderAt,err := zip.NewReaderAt(barReaderAt)

   for _, fBar := range barZipReaderAt.File {  
   xReader, err :=  fBar.Open()
   // read x.txt using io.ReadCloser
   }
}
}

// tar.gz containing zip example
// foo.tar.gz contains: bar.zip
//   bar.zip contains: x.txt
//

f, err := os.Open("foo.tar.gz")
gzReaderAt, err := gzip.OpenReaderAt(f)

tarReader := tar.NewReaderAt(gzf)

for {
header, err := tarReader.Next()

if err == io.EOF {
break
}

switch header.Typeflag {
case tar.TypeDir:
continue
case tar.TypeReg:
raBarZip,err := zip.NewReaderAt(tarReader)
for _, fBar := range rBarZip.File {  
rx, err :=  fBar.Open()
// read x.txt using io.ReadCloser
   }
}

This proposal may be slightly or very off. I am not tied to it. But I am 
looking for a solution that supports the use case of being able to 
recursively drill down into zip and tar(.gz|.bz2) on-disk archives that 
contain an unknown number and depth of contained zip and tar(.gz|.bz2) 
archives, without having to read the zip file entirely into memory, like 
the above stackoverflow example (which I have tried).

Thanks,
Glen  :-)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1108e024-5e0b-437e-b75e-7db64d0e176an%40googlegroups.com.

[go-nuts] New methods on imported structs: how do I do this with complex external structs

[Glen Newton]

Hello,

Context: looked at: 
https://stackoverflow.com/questions/28800672/how-to-add-new-methods-to-an-existing-type-in-go
  

as well as the discussion on this issue:
""cannot define new methods on non-local type ___" is not always an error 
#31631"
https://github.com/golang/go/issues/31631

Often, the imported structs are inter-related, and you might be wanting to 
add a method to a struct that is contained in another external struct 
directly or as an array. So the solution suggested both in the 
stackoverflow response, and in the github discussion appear to me (please 
correct me if I am mistaken) not help with more complex external struct 
inter-relationships.

Please take a look at my example in the playground: 
https://play.golang.org/p/c9jJvdVTU5l

I would like to add a method to an external struct, then add an array of 
these to a struct that contains an array of the original external struct. 
Using either a new type or embedding a type will not work, creating either 
a:
> cannot use make([]bEmbedded, 10) (type []bEmbedded) as type []B in 
assignment
or
> cannot use make([]bType, 10) (type []bType) as type []B in assignment

Am I going about this the wrong way?  :-)  I would appreciate help on this.

[Apologies if this has already been asked and resolved]

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3a617351-fafd-4302-a1ff-d5368f99c5c3n%40googlegroups.com.

Re: [go-nuts] Get name of struct implementing interface using interface method, reflection

[Glen Newton]

Thanks!

Oh no, when I try that here: https://play.golang.org/p/RV-S4MJWYUi
Did not work: similar results: instead of:

BaseFoo
BaseFoo
BaseFoo
BaseFoo

now I get:
main.BaseFoo
main.BaseFoo
main.BaseFoo
main.BaseFoo

Other suggestions?  :-)

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/53e2c85d-9417-4108-abe3-da436b146fb7%40googlegroups.com.

[go-nuts] Get name of struct implementing interface using interface method, reflection

[Glen Newton]

Hello,

I would like to get the name of the struct implementing an interface (using 
a method in the interface). I could just have the MyInterface.StructName() 
method in each implementing struct which just did a return "myname" but I 
would prefer to do this using reflection and without having to replicate 
the same method in all implementations,  I have tried using an embedded 
struct here[1], but it prints the name of the embedded struct. Is there a 
way to do this? 

Why: I have multiple implementations and want to be able to record which 
implementation was used.

Thanks,
Glen


[1]https://play.golang.org/p/CiXIzzLBHOI

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/55161a31-7f8f-4aae-82d0-6b19096cff9b%40googlegroups.com.

[go-nuts] OPI: IBM Job: Go Compiler Developer on IBM's Z Enterprise

[Glen Newton]

Interesting. I just stumbled on it (I have no affiliations).

https://careers.ibm.com/ShowJob/Id/850903/Go%20Compiler%20Developer














*"The IBM Compiler development team is looking to hire Open Source software 
developers with a passion for Golang to join our mission in developing a 
world class offering on IBM platforms. The IBM Toronto Lab is the 
compilation hub within IBM, known for developing world-class compilers and 
runtimes. You will get an opportunity to work alongside technical experts 
in both compilation and cloud development in helping to accelerate our 
mainframe’s journey to the cloud.As a developer on this team, you will be 
responsible for working with internal teams and with the Open Source 
Community for designing, developing and testing the Golang compiler and 
libraries. You will also be responsible for optimizing and tuning that code 
for the IBM's Z Enterprise platform and for ensuring that those features 
are accepted upstream by the Golang community. The main responsibilities of 
the team include:- enable the latest version of Golang compiler on IBM's Z 
Enterprise platform- problem determination, problem recreation, providing 
fixes, validating fixes and getting fixes accepted by the Golang 
Community.- optimizing and tuning that code for IBM's Z Enterprise 
platform- Collaborate with members of the container development and 
deployments teams situated across the globe on a daily basis.- Influence 
the design of future generation of hardware and OS enhancements thorough 
collaboration with IBM hardware/OS teams.- Software developers on the team 
will participate in all aspects of the software development process, from 
design to implementation, from testing to product delivery and support. 
Ideal candidates should feel comfortable working in an agile software 
development environment. The job is located at the IBM Canada Software Lab 
in Markham Ontario, one of the largest Research and Development (R&D) 
facilities in Canada."*

Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/944c3e5a-c054-4b5d-9758-2c1028785b8b%40googlegroups.com.

Re: [go-nuts] Re: Temporarily allow unused variables

[Glen Newton]

I use the "if false{..." extensively as described above.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go Performance

[Glen Newton]

I would also be interested in the memory comparison.

Glen

On Tuesday, October 30, 2018 at 1:03:13 PM UTC, T L wrote:
>
> OP really should compare the memory consumptions of the Java version and 
> Go version. 
>
> On Monday, October 29, 2018 at 4:53:52 PM UTC-4, robert engels wrote:
>>
>> Hello Gophers,
>>
>> I’ve been chastised in the past regarding some of my Go performance tests 
>> - using Java as a baseline. In most of these micro-benchmarks Go has 
>> performed more poorly than expected.
>>
>> So, I ported nats (https://nats.io) message broker (which is written in 
>> Go) to Java. It is available at github.com/robaho/jnatsd
>>
>> This was an ideal test case, as only the server broker was ported, and 
>> all of the client tools and benchmarks could be run unmodified.
>>
>> So, using the Java based broker (under OpenJDK11 with Shenandoah) and the 
>> nats bench with defaults:
>>
>> without SSL, 3200k msgs per second
>> with SSL, 900k msgs per second
>>
>> using the Go based server (under Go 1.11) and the same test parameters:
>>
>> without SSL, 5400k msgs per second
>> with SSL, 2600k msgs per second
>>
>> The latency numbers are much closer, but notice the much higher max in 
>> Java (attributed to larger GC pauses). It is expected that in the above 
>> throughput tests, more GC cycles, thus a much larger penalty.
>>
>> Java: 
>>   latency avg 191214 min 167079 max 936848 ns
>> Go:
>>   latency avg 188338 min 158997 max 588876 ns
>>
>> So, Go shows VERY, VERY impressive performance numbers in a real-world 
>> application, and aligns with my thoughts that Go is ideally suited to 
>> replacing systems level software currently written in C/C++.
>>
>> Some notes, the Java implementation uses some custom string processing 
>> classes to improve performance, but does not use async IO (mainly since 
>> doing so with SSL is a huge PITA). The Go server is also FAR more robust in 
>> terms of ‘handling slow consumers, handling lots of clients, etc.”
>>
>> I might try modifying the Java to use NIO direct buffers and channels to 
>> observe any improved performance.
>>
>>
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Why isn't os.File an interface?

[Glen Newton]

Thanks!
Apologies for not doing a quick search for this.

Thanks,
Glen

On Monday, April 9, 2018 at 2:40:19 PM UTC-4, Glen Newton wrote:
>
> Hello,
>
> I was wondering why os.File is a struct and not an interface. It does not 
> expose anything except methods.
>
> I was wanting to make a mock os.File for testing and other purposes, but 
> it not being an interface makes this not easy. Unless I am missing 
> something here.
>
> Thanks,
> Glen
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Why isn't os.File an interface?

[Glen Newton]

Hello,

I was wondering why os.File is a struct and not an interface. It does not 
expose anything except methods.

I was wanting to make a mock os.File for testing and other purposes, but it 
not being an interface makes this not easy. Unless I am missing something 
here.

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: golang with single-page HTML.

[Glen Newton]

Try https://github.com/icza/gowut

https://gowut-demo.herokuapp.com/show

Glen

On Wednesday, January 24, 2018 at 4:24:51 PM UTC-5, matthe...@gmail.com 
wrote:
>
> One way is the server still serves all of those endpoints, but instead of 
> returning a full HTML page they each just return data. The single page 
> state depends on what data is received when each is queried. Using a 
> template for the single page is a way to set an initial state.
>
> jQuery has easy to use AJAX functions (GET with arguments encoded into the 
> URL and POST with arguments as part of the request body). Websockets are 
> another way to pass data.
>
> This pattern in the regular Go http handlers goes far with a JavaScript 
> client:
>
> js, err := json.Marshal(output)
> if err != nil {
> panic(err.Error())
> }
> w.Header().Set("Content-Type", "application/json")
> w.Write(js)
>
> Matt
>
> On Tuesday, January 23, 2018 at 8:13:06 PM UTC-6, jacktee...@gmail.com 
> wrote:
>>
>> i have a few .go programs that use the old /, /index, /about, /login, 
>> /signup etc. I have been trying to use the SPA single-page HTML. But I 
>> cannot figure out how to use SPA with my golang modules. Anyone have any 
>> simple examples of this? If I had my druthers, I would try use angular5 for 
>> screen handling, but AJAX is fine if someone has an example. 
>> New to the group. Thanks in advance for any advice or help!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Web Framework for Beginners

[Glen Newton]

I am not going to be popular expressing this, but I really really really 
like gowut: https://github.com/icza/gowut

Just write (drop-dead simple) Go code for SPA, AJAX handled transparently, 
no fiddling with Javascript.

See: Gowut - Showcase of Features  
https://gowut-demo.herokuapp.com/show

Downside: all client events roundtrip to server.


Glen

On Wednesday, January 24, 2018 at 6:01:06 PM UTC-5, Michael Masouras wrote:
>
> I second what Shawn said, I just use the standard library. Whenever I need 
> something not quite covered (or an easier abstraction), I use gorilla 
> .t goi
>
> On Wed, Jan 24, 2018 at 6:16 AM, dc0d 
> > wrote:
>
>> gobuffalo  has a nice workflow (I've used it for 
>> two in-house apps).
>>
>>
>> On Wednesday, January 24, 2018 at 5:20:05 PM UTC+3:30, pradam wrote:
>>>
>>> Hi Gophers,
>>>
>>> As for Newbie to Go.I am excited to learn a *web framework* but I 
>>> don't know where to start please help me out by suggesting a good framework 
>>> for beginners.
>>>
>>> thank you,
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [ANN] Go Jupyter kernel and an interactive REPL

[Glen Newton]

Wow! This is great and positions Go better in the 'data science' world!

Thanks,
Glen

On Tuesday, January 16, 2018 at 8:53:39 AM UTC-5, Yu Watanabe wrote:
>
> Hi Gophers,
>
> I developed a new Go's Jupyter Notebook  kernel and 
> REPL environment.
> I would like to announce it to golang users and hear feedback about it.
>
> https://github.com/yunabe/lgo - Go (golang) Jupyter Notebook kernel and an 
> interactive REPL 
>
> Features:
> 1. You can write and execute Go code interactively on browsers and CUI 
> console.
> 2. Full Go language spec support.
> 3. Code completion and code inspection
> 4. Display images, HTML, JavaScript, SVG etc..
> 5. Code cancellation
>
> Thanks to mybinder.org, we can try example notebooks without installation 
> from this link:
> https://mybinder.org/v2/gh/yunabe/lgo-binder/master?filepath=basics.ipynb
>
> If you think it's interesting, please go through the instructions in 
> README.md and install it in your Go environment.
>
> Thanks,
> yunabe
>
> Screenshots:
>
>
>
> 
>
> ** code inspection (Shift-Tab) **
>
> 
>
> ** code completion (Tab) **
>
> 
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Calling the Go compiler inside of tests

[Glen Newton]

YES! This looks like it is exactly what I needed!
Thanks!

Glen

On Friday, December 15, 2017 at 3:55:51 PM UTC-5, Ian Cottrell wrote:
>
> You could just use the go/types package to typecheck the result instead of 
> compiling it, depending on exactly what you want to check that may be 
> enough.
>
>
> On Fri, Dec 15, 2017 at 8:37 AM Tamás Gulácsi  > wrote:
>
>> When you invoke "go test", it compiles a test binary and calls that.
>> Why can't you just repeat that?
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Calling the Go compiler inside of tests

[Glen Newton]

Hello,

Apologies if this has already been discussed.

I have a go tool (chidley - https://github.com/gnewton/chidley) that 
converts xml to Go struct code by examining a sample file (perhaps very 
large) of the XML (It can also generate Java JAXB code...).
Production examples of generated structs:

   - https://github.com/gnewton/pubmedstruct/blob/master/pubmed.go
   - https://github.com/gnewton/struct2dot
   - https://github.com/gnewton/gomesh2016/blob/master/desc2016.go
   

[BTW, it was the first Go thing I made so it is a little...peculiar. Not 
idiomatic and not well aligned with Go interfaces, concurrency, etc.]

My question is about testing: this application generates Go code. I would 
like to have a test that is successful if the generated Go code compiles. 
As I am using a template for the generated code, and the Go code in the 
template compiles correctly, the only place where compilation would fail 
would be if the Go structs generated by chidley were not correct. It should 
never generate structs that do not compile and I want to test for this. I 
have a set of complicated/problematic XML that I would like to walk through 
via testing to make sure my changes to the application do not cause 
regression issues. 

I could do this just by invoking the Go compiler using os.exec and checking 
the return code. But I was wondering if there were better ways of doing 
this that was 'inside' of the testing framework or some other way besides 
going to the host os.

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Just launched a new Go Community Site

[Glen Newton]

Suggest displaying timestamp of entries. Very annoying (read 'very un-useful') 
to look through such a list without a published date/time...

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: production PDF to text service

[Glen Newton]

Could you list the 2 packages?

Glen

On Sunday, June 18, 2017 at 6:27:40 PM UTC-4, svj...@gmail.com wrote:
>
> Can someone with experience with the problem suggest a best package for 
> building a production PDF to text conversion service in Go? I am working 
> with Go and Google App Engine. A cursory Google search brings up two 
> packages, the first considerably more popular but with external 
> dependencies I would like to avoid.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: chidley: XML to Go structs

[Glen Newton]

Without your XML file I cannot figure out the issue.
If you are OK with sharing your XML file, please post it with the issue at: 
https://github.com/gnewton/chidley/issues

Thanks,
Glen


On Wednesday, June 14, 2017 at 2:54:28 PM UTC-4, shrinkhala singhania wrote:
>
> Hi,
>
> I tried running my XML file and throws this error:
>
>
> chidley$chidley -W xml/openconfig.xml > examples/oc/openconfig1.go
>
> chidley$cd examples/oc/
>
> oc$go build
>
># github.com/gnewton/chidley/examples/oc
>
>   ./openconfig1.go:3361: Chiroot redeclared in this block
>
>previous declaration at ./openconfig1.go:213
>
> Regards,
> Shrinkhala
>
> On Wed, Jun 14, 2017 at 2:25 PM, Shrinkhala Singhania <
> singhani...@gmail.com > wrote:
>
>> That worked. Thanks.
>>
>> Regards,
>> Shrinkhala
>>
>> On Wed, Jun 14, 2017 at 2:22 PM, Ian Lance Taylor > > wrote:
>>
>>> On Wed, Jun 14, 2017 at 11:08 AM,  > 
>>> wrote:
>>> > I have this in my bash_profile:
>>> >
>>> > export PATH=/usr/local/bin:/usr/local/sbin:$PATH
>>> >
>>> > export GOPATH = /Users/ssinghania/go/
>>> >
>>> > export CLASSPATH=".:/usr/local/lib/antlr-4.6-complete.jar:$CLASSPATH"
>>>
>>> OK, so do what I suggested.  You need something like
>>>
>>> export PATH=/usr/local/bin:/usr/local/sbin:/Users/ssinghania/go/bin:$PATH
>>>
>>> Ian
>>>
>>>
>>> > On Wednesday, June 14, 2017 at 1:53:53 PM UTC-4, Ian Lance Taylor 
>>> wrote:
>>> >>
>>> >> On Wed, Jun 14, 2017 at 10:19 AM,   wrote:
>>> >> >
>>> >> > To install chidley, I did the following:
>>> >> >
>>> >> > go get github.com/gnewton/chidley
>>> >> >
>>> >> > This worked. However, issue on
>>> >> >
>>> >> >
>>> >> > chidley -W xml/test1.xml > examples/test/ChildTest1.go
>>> >> >
>>> >> > -bash: chidley: command not found
>>> >> >
>>> >> > How do I fix this?
>>> >>
>>> >> Put $GOPATH/bin on your PATH.  If you haven't set $GOPATH, the default
>>> >> is $HOME/go.
>>> >>
>>> >> Ian
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google 
>>> Groups
>>> > "golang-nuts" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send 
>>> an
>>> > email to golang-nuts...@googlegroups.com .
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: filago: Monitor a Linux process' opening and closing of files, including Unix pipes and sockets

[Glen Newton]

The functionality filago provides can be implemented with truss or strace. 

filago offers a simpler (to many) way of getting to this information, and 
can provide it in json as well.

Thanks,
Glen


On Monday, June 5, 2017 at 3:25:20 PM UTC-4, Simon Ritchie wrote:
>
> Does this have some advantage over truss (or under Linux, strace)?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: filago: Monitor a Linux process' opening and closing of files, including Unix pipes and sockets

[Glen Newton]

Sorry about that: link to github: https://github.com/gnewton/filago

Thanks,
Glen

On Friday, June 2, 2017 at 11:56:34 PM UTC-4, Glen Newton wrote:
>
> Monitor open files (via /proc/pid) of a process. Lists time when it first 
> sees the file open (which may be later than when the file actually opened, 
> especially at startup of filago), and when it is no longer open. Stop 
> running when target process ends.
>
> This includes anon_inode and tcp and unix sockets.
>
>
> *Only works on Linux or OSes supporting /proc in this manner.*
>
>
> Usage of ./filago:
>   -d uint
>   Time granularity for checking files, in milliseconds (default 100)
>   -j  Output json (complete json per line)
>   -l  Turn on hostname lookup (default is a "-"
>   -r  Show only real files, i.e. no pipes, sockets, etc.
>
>
> Default simple text output as well as json output containing more 
> extensive information.
>
>
> Example text output on Firefox process:
>
>
> $ ./filago -l -j 3737
>
> 2017-05-07T12:40:40.663016-04:00 open socket:[416916] tcp 
> 198.252.206.25:443 stackoverflow.com. 
> 2017-05-07T12:40:40.663016-04:00 open anon_inode:[eventpoll] 
> 2017-05-07T12:40:40.663016-04:00 open socket:[511791] unix - 
> 2017-05-07T12:40:40.663016-04:00 open 
> /home/bsmith/.cache/event-sound-cache.tdb.9bf539dba0e34f7aaf456bd844b6826e.x86_64-redhat-linux-gnu
>  
>
> 2017-05-07T12:40:40.663016-04:00 open socket:[1265977] unix - 
> 2017-05-07T12:40:40.663016-04:00 open socket:[1574737] tcp 
> 173.194.175.189:443 qs-in-f189.1e100.net. 
> 2017-05-07T12:41:21.110231-04:00 close socket:[1630334] tcp 
> 52.84.139.212:80 server-52-84-139-212.yto50.r.cloudfront.net. 
> 2017-05-07T12:42:35.083678-04:00 open socket:[1635598] 
> 2017-05-07T12:42:35.155006-04:00 close socket:[1635598] 
> 2017-05-07T12:42:35.155006-04:00 open socket:[1630691] tcp 
> 172.217.6.229:443 lga25s55-in-f5.1e100.net.
>
>
> Example json output (complete json per line) on Firefox process:
>
>
> $ ./filago -l -j 3737
> {"filename":"socket:[95066]","type":"unix","socket_info":{"unix_socket":{"num":"8803e6c99f80","refcount":"0003","protocol":"","flags":"","type":"0001","st":"03","inode":95066}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
> {"filename":"socket:[95067]","type":"unix","socket_info":{"unix_socket":{"num":"8803e6c99c00","refcount":"0003","protocol":"","flags":"","type":"0001","st":"03","inode":95067}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
> {"filename":"pipe:[95068]","type":"pipe","status":"open","mod_time":"2017-06-02T23:20:10.616170585-04:00"}
> {"filename":"/home/bsmith/install/firefox/omni.ja","type":"file","status":"open","mod_time":"2017-05-27T23:24:59.740556075-04:00"}
> {"filename":"/home/bsmith/install/firefox/browser/omni.ja","type":"file","status":"open","mod_time":"2017-05-27T23:25:00.173556279-04:00"}
> {"filename":"pipe:[95069]","type":"pipe","status":"open","mod_time":"2017-06-02T23:19:57.685144488-04:00"}
> {"filename":"pipe:[110949]","type":"pipe","status":"open","mod_time":"2017-06-02T23:19:57.685144488-04:00"}
> {"filename":"socket:[95070]","type":"other","status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
> {"filename":"pipe:[102268]","type":"pipe","status":"open","mod_time":"2017-06-02T23:20:13.688176034-04:00"}
> {"filename":"/home/bsmith/.mozilla/firefox/uqsr6u0q.default/extension-data/ublock0.sqlite","type":"file","status":"open","mod_time":"2017-06-02T23:19:09.638842227-04:00"}
> {"filename":"socket:[110951]","type":"unix","socket_info":{"unix_socket":{"num":"880403259c00","refcount":"0

[go-nuts] filago: Monitor a Linux process' opening and closing of files, including Unix pipes and sockets

[Glen Newton]

Monitor open files (via /proc/pid) of a process. Lists time when it first 
sees the file open (which may be later than when the file actually opened, 
especially at startup of filago), and when it is no longer open. Stop 
running when target process ends.

This includes anon_inode and tcp and unix sockets.


*Only works on Linux or OSes supporting /proc in this manner.*


Usage of ./filago:
  -d uint
Time granularity for checking files, in milliseconds (default 100)
  -jOutput json (complete json per line)
  -lTurn on hostname lookup (default is a "-"
  -rShow only real files, i.e. no pipes, sockets, etc.


Default simple text output as well as json output containing more extensive 
information.


Example text output on Firefox process:


$ ./filago -l -j 3737

2017-05-07T12:40:40.663016-04:00 open socket:[416916] tcp 
198.252.206.25:443 stackoverflow.com. 
2017-05-07T12:40:40.663016-04:00 open anon_inode:[eventpoll] 
2017-05-07T12:40:40.663016-04:00 open socket:[511791] unix - 
2017-05-07T12:40:40.663016-04:00 open 
/home/bsmith/.cache/event-sound-cache.tdb.9bf539dba0e34f7aaf456bd844b6826e.x86_64-redhat-linux-gnu
 

2017-05-07T12:40:40.663016-04:00 open socket:[1265977] unix - 
2017-05-07T12:40:40.663016-04:00 open socket:[1574737] tcp 
173.194.175.189:443 qs-in-f189.1e100.net. 
2017-05-07T12:41:21.110231-04:00 close socket:[1630334] tcp 
52.84.139.212:80 server-52-84-139-212.yto50.r.cloudfront.net. 
2017-05-07T12:42:35.083678-04:00 open socket:[1635598] 
2017-05-07T12:42:35.155006-04:00 close socket:[1635598] 
2017-05-07T12:42:35.155006-04:00 open socket:[1630691] tcp 
172.217.6.229:443 lga25s55-in-f5.1e100.net.


Example json output (complete json per line) on Firefox process:


$ ./filago -l -j 3737
{"filename":"socket:[95066]","type":"unix","socket_info":{"unix_socket":{"num":"8803e6c99f80","refcount":"0003","protocol":"","flags":"","type":"0001","st":"03","inode":95066}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"socket:[95067]","type":"unix","socket_info":{"unix_socket":{"num":"8803e6c99c00","refcount":"0003","protocol":"","flags":"","type":"0001","st":"03","inode":95067}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"pipe:[95068]","type":"pipe","status":"open","mod_time":"2017-06-02T23:20:10.616170585-04:00"}
{"filename":"/home/bsmith/install/firefox/omni.ja","type":"file","status":"open","mod_time":"2017-05-27T23:24:59.740556075-04:00"}
{"filename":"/home/bsmith/install/firefox/browser/omni.ja","type":"file","status":"open","mod_time":"2017-05-27T23:25:00.173556279-04:00"}
{"filename":"pipe:[95069]","type":"pipe","status":"open","mod_time":"2017-06-02T23:19:57.685144488-04:00"}
{"filename":"pipe:[110949]","type":"pipe","status":"open","mod_time":"2017-06-02T23:19:57.685144488-04:00"}
{"filename":"socket:[95070]","type":"other","status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"pipe:[102268]","type":"pipe","status":"open","mod_time":"2017-06-02T23:20:13.688176034-04:00"}
{"filename":"/home/bsmith/.mozilla/firefox/uqsr6u0q.default/extension-data/ublock0.sqlite","type":"file","status":"open","mod_time":"2017-06-02T23:19:09.638842227-04:00"}
{"filename":"socket:[110951]","type":"unix","socket_info":{"unix_socket":{"num":"880403259c00","refcount":"0003","protocol":"","flags":"","type":"0001","st":"03","inode":110951}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"anon_inode:[eventfd]","type":"anon_inode","status":"open","mod_time":"2017-06-02T22:20:43.18337-04:00"}
{"filename":"anon_inode:inotify","type":"anon_inode","status":"open","mod_time":"2017-06-02T22:20:43.18337-04:00"}

{"filename":"socket:[109319]","type":"tcp","socket_info":{"inode":109319,"tcp_socket":{"rem_hostname":"a23-63-227-177.deploy.static.akamaitechnologies.com.","sl":4,"local_port":58795,"remote_port":80,"st":1,"tr":1,"uid":1000,"tx_queue":296,"tm_when":21,"extra":"3","rem_address":"23.63.227.177","local_address":"192.168.0.101"}},"status":"close","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"socket:[106227]","type":"tcp","socket_info":{"inode":106227,"tcp_socket":{"rem_hostname":"a23-63-227-177.deploy.static.akamaitechnologies.com.","sl":1,"local_port":58796,"remote_port":80,"st":1,"tr":1,"uid":1000,"tx_queue":296,"tm_when":21,"extra":"3","rem_address":"23.63.227.177","local_address":"192.168.0.101"}},"status":"open","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"socket:[106227]","type":"tcp","socket_info":{"inode":106227,"tcp_socket":{"rem_hostname":"a23-63-227-177.deploy.static.akamaitechnologies.com.","sl":1,"local_port":58796,"remote_port":80,"st":1,"tr":1,"uid":1000,"tx_queue":296,"tm_when":21,"extra":"3","rem_address":"23.63.227.177","local_address":"192.168.0.101"}},"status":"close","mod_time":"1969-12-31T19:00:00-05:00"}
{"filename":"socket:[111430]","type":"tcp","socket_info":{"inode":111430,"tcp_socket":{"rem_hostname":

[go-nuts] Proposal: Tags on Interface types

[Glen Newton]

I would like to propose allowing annotations (tags) on Interface types. 
Only a subset of an Interfaces methods could be tagged. This subset is 
defined as methods that have a single return type and take no parameters 
(as a best practice, these methods should not change any state and be 
thread safe...). 

So let's use the 'Go by example Interface', found here:
https://gobyexample.com/interfaces
and
https://play.golang.org/p/313UebA3rD

I have an altered version of this, where I have capitalized the methods of 
interest and added tags: https://play.golang.org/p/50oEQHeWVZ

// _Interfaces_ are named collections of method
// signatures.

package main

import "fmt"
import "math"

// Here's a basic interface for geometric shapes.
type geometry interface {
Area() float64 `json:"area"`
Perim() float64 `json:"perimeter"`
}

// For our example we'll implement this interface on
// `rect` and `circle` types.
type rect struct {
width, height float64
}
type circle struct {
radius float64
}

// To implement an interface in Go, we just need to
// implement all the methods in the interface. Here we
// implement `geometry` on `rect`s.
func (r rect) Area() float64 {
return r.width * r.height
}
func (r rect) Perim() float64 {
return 2*r.width + 2*r.height
}

// The implementation for `circle`s.
func (c circle) Area() float64 {
return math.Pi * c.radius * c.radius
}
func (c circle) Perim() float64 {
return 2 * math.Pi * c.radius
}

// If a variable has an interface type, then we can call
// methods that are in the named interface. Here's a
// generic `measure` function taking advantage of this
// to work on any `geometry`.
func measure(g geometry) {
fmt.Println(g)
fmt.Println(g.Area())
fmt.Println(g.Perim())
}

func main() {
r := rect{width: 3, height: 4}
c := circle{radius: 5}

// The `circle` and `rect` struct types both
// implement the `geometry` interface so we can use
// instances of
// these structs as arguments to `measure`.
measure(r)
measure(c)
}

What happens? When a tag processor is engaged on an interface containing 
tags, instead of using value of the fields of a struct that are tagged, it 
takes the value returned by the invocation of the methods that are tagged 
(it invokes the method and uses the (single) value returned). 

So what does this give us? It means I can have a list of type geometry 
(interface) that contains both circle and square structs under the hood, 
and I can walk through the list and (for example) serialize them, using the 
serialization defined in the interface

It gives a bit of generic feel to things, at least from a tags perspective. 
[Sorry, I said the 'G' word]

I guess reflect.Method  would need 
a Tag StructTag  like that in 
reflect.StructField. 

I think this would be backward compatible, and just a handful of things 
would need to be updated in reflect (from my limited knowledge) as well as 
some minor changes to the compiler (??) and likely other things I don't 
know about, like various tool chain stuff, etc, ??.

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How do I convert a byte into BigEndian or LittleEndian?

[Glen Newton]

Thanks!  :-)

On Monday, February 27, 2017 at 4:45:19 PM UTC-5, Glen Newton wrote:
>
>
>
> Given a byte b, how do I convert it to a byte of particular endianness?
> It is not clear to me looking at "encoding/binary"
>
> (I am assuming that golang byte endianness varies across architectures. 
> If this is wrong please educate me). :-)
>
> Thanks,
> Glen
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How do I convert a byte into BigEndian or LittleEndian?

[Glen Newton]

Ah, I've been thinking about this issue too much and have confused myself.

And, I have expressed it as an XY Problem!  :-(

My real question should have been: if I write a byte to a file on one 
architecture, will the byte be the same on all other architectures (i.e. 
386, amd64, arm, s390x, ppc64le) if read by a Go program on them?

Sorry for my confusion.  :-)

Thanks,
Glen

On Monday, February 27, 2017 at 4:45:19 PM UTC-5, Glen Newton wrote:
>
>
>
> Given a byte b, how do I convert it to a byte of particular endianness?
> It is not clear to me looking at "encoding/binary"
>
> (I am assuming that golang byte endianness varies across architectures. 
> If this is wrong please educate me). :-)
>
> Thanks,
> Glen
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How do I convert a byte into BigEndian or LittleEndian?

[Glen Newton]

Ack! Sorry.

I was referring to *bit endianness* 
https://en.wikipedia.org/wiki/Bit_numbering

On Monday, February 27, 2017 at 4:45:19 PM UTC-5, Glen Newton wrote:
>
>
>
> Given a byte b, how do I convert it to a byte of particular endianness?
> It is not clear to me looking at "encoding/binary"
>
> (I am assuming that golang byte endianness varies across architectures. 
> If this is wrong please educate me). :-)
>
> Thanks,
> Glen
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] How do I convert a byte into BigEndian or LittleEndian?

[Glen Newton]

Given a byte b, how do I convert it to a byte of particular endianness?
It is not clear to me looking at "encoding/binary"

(I am assuming that golang byte endianness varies across architectures. If 
this is wrong please educate me). :-)

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Could Go http serve somehow use Apache modules? Interested in modsecurity in particular...

[Glen Newton]

Hello,

I was wondering if Apache modules could somehow be exposed and used by Go 
http.

My motivation is that ModSecurity is pretty attractive: 
https://modsecurity.org/

I know the default config for Go web is to put it behind Apache or nginx, 
but recent discussions 
https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/ suggest 
that directly exposing Go http might become more common.

Alternatively, is there something out there in Go Land comparable to the 
above modsecurity? Something that could handle the OWASP ModSecurity Core 
Rule Set 3.0 https://www.modsecurity.org/crs/ would be what I was looking 
for.

Thanks,
Glen  :-)


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Mobile + camera

[Glen Newton]

Hi Elias,

Thanks for the update, I will try some things out.

My use case is pretty simple: a go based web server on a very low end 
Android device where all it does is take a picture with the camera and 
serves it for each GET / on the web server.
A Go based solution would have a lower footprint for this device (unless I 
am mistaken).

Thanks,
Glen

On Friday, September 23, 2016 at 11:51:31 AM UTC-4, Elias Naur wrote:
>
> Hi Glen,
>
> The proposal for accessing Android APIs directly has landed: 
> https://github.com/golang/go/issues/16876. In theory, that should be 
> enough for accessing the camera from Go. In practice, the new features are 
> very recent and you might encounter problems along the way. Feel free to 
> report any issues you encounter and CC them to me :)
>
>  - elias
>
> On Friday, September 23, 2016 at 5:16:08 PM UTC+2, Glen Newton wrote:
>>
>> Are there any updates on this topic, i.e. can I use Go to talk to the 
>> Android camera?
>>
>> Thanks,
>> Glen
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Mobile + camera

[Glen Newton]

Are there any updates on this topic, i.e. can I use Go to talk to the Android 
camera?

Thanks,
Glen

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.