My guess is that most applications are decoding 25gb json files with any regularity. Even transferring 25 GB over the fastest of networks takes 20 secs? So that reduces the cost to less than 10%???
The test isn’t doing anything with the decoded json - maybe Go is 1000% faster in that.
This is either a troll post or by someone that needs more education in performance monitoring. Sorry. On Mar 8, 2024, at 10:25 PM, Mike Schinkel <m...@newclarity.net> wrote:
Hi Robert,
Thanks for the reply.
I decided to run a quick benchmark so I compared Go 1.22's json.UnMarshal(any) using an any type with PHP 8.3.2's json_decode()
and found that PHP's option is 85% faster for a ~25Mb JSON file. Optimizing it with a struct I was able to get Go down to being only about 25% slower than PHP's json_decode(). Note that they are very rough benchmarks, probably only valid in gauging magnitude, not specific performance. I wrote up the results and provided the code in this gist for anyone who might be interested. For me, the takeaway is that if you really need fast JSON unmarshalling you probably need to look to one of the 3rd party JSON parser packages.
Whether or not the extra time it takes Go to parse JSON vs. PHP is actually relevant for her use-case remains to be seen given the potential order-of-magnitude difference in web request latency. Still, given that she seems to be controlling for that web request latency by comparing her PHP API to her Go API would indicate — sans issues like buffering or slow Go middleware — that the performance of JSON parsing made indeed by one of the reasons she is seeing a difference.
Pragya, it would be really nice if you could follow up to close the loop and let us know what the actual bottleneck was and how you ended up solving it.
-Mike
On Friday, March 8, 2024 at 9:58:26 PM UTC-5 Robert Engels wrote:
Related, 30k of anything is largely meaningless for modern hardware in terms of cpu processing - where billions of operations a second is the norm.
The point I was trying to make - maybe unclear - is that it is doubtful any raw performance differences between PHP and Go in terms of json encoding/decoding would be noticeable- unless it was a hft/hpc system which would be doubtful given that the original system was PHP.
So if there is detectable difference in performance it is almost certainly related to IO management concerns (buffering, window sizes, etc)
Without detailed measurements along with how they were measured it’s hard to know with any certainty- so this was a Large guess to help you know where to look. Hi Robert,
I am now confused. While your original reply to this thread made perfect sense to me, your follow up for the purposes of clarification had the opposite effect, at least for me. I am not clear the point you are trying to make.
You mention the difference between compiled and interpreted and point out that typically there is not a large performance difference — where I assume you were referring to use of standard library functions in the interpreted language such as json_decode() in PHP. That is of course often true and I concur. (Of course, try writing a full JSON parser in pure PHP and we'll find a big difference in performance with a compiled language like Go.)
But I am not seeing why that point is relevant in the thread because she was not asking "Why is PHP code slower than Go code?" in which case I would understand why you chose to make that distinction.
You also mention that IO performance (and design(?)) over the web is typically much more relevant performance-wise, on which I also concur, but given that both her PHP and her Go API endpoints presumably had much the same web-based latency and response time concerns, I'm struggling to understand why it was relevant to state it in this context. Of course your point simply could have been not using a buffered output stream, but you already mentioned, and your follow up did not make that connection clear.
So, can you please help me better understand the point you were trying to make? As a follow up to my reply were you trying to imply that illustrating ways to optimize JSON parsing was something you felt I should not have posted in response to her question? That feels like in might have been your intent, but I could certainly have misinterpreted and if so would prefer to know rather than wrongly assume.
-Mike On Friday, March 8, 2024 at 8:28:38 PM UTC-5 Robert Engels wrote:
Just to be clear for others - from a raw cpu performance perspective when looking at a typical application in whole - there is very little performance difference between compiled and even GC platforms- interpreted can lag behind a bit - but in almost all cases over the web it is IO performance/design that matters the most (most middleware connects to other middleware, etc).
Some HFT/HPC systems deviate from this but by the description and reference to the previous implementation I doubt that is the case here. Hi Pragya,
While Robert Engles is probably correct in identifying your bottleneck, if it does turn out to be slow JSON parsing here are a few things you can look at. 1. You mention you have to use a map because of response keys not being fixed. Be aware that you do not need to create a struct to match the full JSON. You can easily just create a flyweight struct which is a subset if for your use-case containing only the specific parts you need, for example. 2. If you optionally need to parse portions of the JSON, but not always, you can use json.RawMessage to capture the properties you don't always need to parse, and then parse them only when you need to. 3. You can also declare an .UnmarshalJSON() method on a struct you are passing to json.Unmarshal(), or on any struct that is a property of your top level object, and then be fully in control of parsing the data, meaning you could combine with (a) json.RawMessage property(s) to part just the non-fixed keys as a map, and only for use-cases when you need to. 4. And finally, if you are willing to venture out from the Go standard library, there are numerous open-source packages that claim to be much faster than the standard library. Usually you want to stick with the Go standard library so other Go developers will be familiar with it and to minimize dependencies that could potentially introduce a bug or security hole. However, if you really need better performance it might be worth the added dependency.
While I cannot validate the performance claims of any of these packages, I can provide you with this list of packages that claim better JSON unmarshalling performance: Benchmarks from authors of some of the packages (so take with a grain of salt):And benchmarks from someone who is not an author of one of those packages: Hope this helps.
-Mike On Friday, March 8, 2024 at 4:15:25 PM UTC-5 Robert Engels wrote:
It is highly unlikely that the Go marshaling is the cause. I’m guessing you are probably not using a buffered output stream.
PHP is written in C but depending on what you are doing it is either a script or calling out to another process (afaik) Hi,
"I am facing an issue in my Go code. I converted my read API from PHP to Go, but the response time of Go API is higher than PHP. We are returning the response in JSON format, and the response contains around 30k keys, which is taking too much time in JSON marshaling. We are adding all the response in map format as we can't use struct because response keys are not fixed and vary from user to user.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/01d857e5-c7ec-431f-9f27-775c2411da05n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6c9e0a59-a2c9-4cc0-9fa3-271838c2cb63n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4E74FADD-0AE3-429B-96F2-E4ACA69F3059%40ix.netcom.com.
|
