Enable compression of API responses

Markuzzz

HTTP has built-in support for compression of HTTP responses with for example gzip.
Currently this does not seem to be supported by the API (but the main website does use it).
Compression leads to a HUGE reduction in bandwith usage.
Small experiments on my end lead to 10x reductions with gzip.
That is not too surprising as JSON typically contains many repeating patterns.

Enabling it should be a matter of changing some configuration setting and will not break existing code (like I said, it is supported by the HTTP protocol).

For more information see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
The Accept-Encoding header in the request lists the accepted encodings by the client.
The Content-Encoding header in the response specifies which encoding was used by the server

Lawrence

I believe that should now be resolved, thank you for bringing that to our attention

Markuzzz

No problem. Thank you for resolving it. I can confirm that compression is enabled, at least for the catalog/availability endpoint. A request for 2 x 4 bricks (BOID 771344) returns 1.1 MB of data which decompresses to 10 MB.

Markuzzz

@Lawrence , it seems that in the same update you have switched off the option to make catalog/availability calls in a bulk request. That is very unfortunate. In effect it is a 50x reduction in capacity. Given that the bandwidth of each catalog/availability call has just been reduced by a factor of 10 due to compression, it also seems a bit unnecessary.

It is also a breaking change for people (like me) that were making these calls.

Lawrence

@Markuzzz unfortunately we had to remove catalog/availability from bulk/batch as we were getting several server alarms related to performance issues yesterday. With your 10x50 requests a second, for queries that can return 1000 lots, this was 500,000 lots / second. The queries would need to be sent in the non-batch normal way, and under the 600 requests / minute which the server can more easily process without affecting other users.

Markuzzz

@Lawrence , I see. I was doing some experiments yesterday fetching the data of common bricks. I made a total of 500 requests at a time (10 bulk requests concurrently) for very common bricks and without specifying a color, so every request would indeed return a lot of lots. It was most certainly not my intention to overload your servers or affect other users.

Would it be an idea to allow batch requests for BOIDs that do specify a color? They typically return much fewer lots. In a sense a request for a BOID with no color specified is already a bit of a bulk request. Without the bulk requests I am inclined to reduce the number of calls I make by not specifying a color, which is a net increase of the load I am causing your servers.

The thing is that a large MOC often contains close to 600 different part-color combinations. It would then take more than a minute to fetch all the data (if the info is no longer in my cache). And that is assuming is 1 user at a time.

Lawrence

We don't use a queue system on API requests so from a server load perspective, it's better to have any expensive requests sent sequentially and spaced apart, rather than in a batch of 50.