Rate limit exceeded keeps failing in production

Private User

12/7/2019 at 4:42 AM

This conversation is linked to this one: https://www.geni.com/discussions/204253

It really looks there is some problem with the Rate limit. I am using access token to call geni API so (ref https://www.geni.com/platform/developer/help/rate_limits?version=1), it should support 40 requests per 10 seconds, so, the limit is 1 request each 0.25 seconds.

I have created an small script in python to test the proble. Essentially, I created this piece of code for running the url calls:

data = None
continue_iteration = True
i=1
while continue_iteration:
print(i,datetime.datetime.now())
data = requests.get(url)
i += 1
value_error = data.json().get("error", {}).get("message", None)
if not ( value_error and ("Rate limit exceeded." in value_error) ): continue_iteration = False

So, my expectatino of output is that all print outs will start with 1, given that the difference between the first call and the next is at least 0.25 seconds.

But this is the result I get:

1 2019-12-07 13:37:34.182276
1 2019-12-07 13:37:35.331122
1 2019-12-07 13:37:36.359418
1 2019-12-07 13:37:36.967325
1 2019-12-07 13:37:37.889597
1 2019-12-07 13:37:38.284527
1 2019-12-07 13:37:38.916835
1 2019-12-07 13:37:39.313022
1 2019-12-07 13:37:40.463302
1 2019-12-07 13:37:40.960288
1 2019-12-07 13:37:41.983683
2 2019-12-07 13:37:42.503397
3 2019-12-07 13:37:43.007764
4 2019-12-07 13:37:43.394448
5 2019-12-07 13:37:43.785502
6 2019-12-07 13:37:44.544111
1 2019-12-07 13:37:45.574507
1 2019-12-07 13:37:45.942273
1 2019-12-07 13:37:47.104166
1 2019-12-07 13:37:48.133591
1 2019-12-07 13:37:48.641903
1 2019-12-07 13:37:49.041802
1 2019-12-07 13:37:49.465901
1 2019-12-07 13:37:50.178529
1 2019-12-07 13:37:50.441913
1 2019-12-07 13:37:50.785818
2 2019-12-07 13:37:51.199413
3 2019-12-07 13:37:51.712037
4 2019-12-07 13:37:52.095940
5 2019-12-07 13:37:52.429748
6 2019-12-07 13:37:52.745671
7 2019-12-07 13:37:53.150062
8 2019-12-07 13:37:53.759839
9 2019-12-07 13:37:54.271167
1 2019-12-07 13:37:54.634139
1 2019-12-07 13:37:55.298050

As you can see, it gives an error of Rate Limit around 10 calls (not always 10), notice that all the calls are done between 0.4-1 second, so below the 0.25 limit

Private User

12/7/2019 at 5:43 AM

in production (www.geni.com) our rate limit is 10-requests every 10-seconds.

Private User

12/8/2019 at 1:49 PM

That changed recently, isn't it? This script was working ok until the last weeks.

Private User

12/8/2019 at 1:51 PM

Or... the computational power has increased driving to faster answers from the server and allowing more requests in 10 seconds and making my software to fail (as not considering this limit)

Private User

12/9/2019 at 5:40 AM

We added more servers a few weeks ago.

Private User

12/9/2019 at 7:36 AM

So it does matches... as the computational capacity was lower, the answer I was receiving before was likely taking more time and making the same script to go slower but below the rate limit...
It looks that now, with a bigger capacity is goes quite faster and I hit the rate limit. Btw, I hit it by just 2 seconds :)

Craig Andrew Miles PRO

3/10/2022 at 4:50 PM

So it seems to me that the Rate Limit has recently been changed from 10 per rate window (10 seconds) to 1 per 10 seconds, judging by the following API response:

x-api-rate-limit:
- '1'
x-api-rate-remaining:
- '0'
x-api-rate-window:
- '10'

This is now completely unworkable when it takes multiple API calls to define a profile and it's relationships. It was bad enough at 1 second per call, requiring about 4 seconds to define a profile.

Multiplying that by 10 means 40 seconds per profile which is ridiculous f=if you are trying to analyse hundreds of profiles.

What is the reason behind this unannounced drastic change?

Mike Stangel G

4/27/2022 at 3:34 PM

Craig Andrew Miles we reduced the new / unapproved app rate limit due to abuse. We'll be happy to approve your app for a higher rate limit if we can get a look at it -- how can we see your app in action?

Craig Andrew Miles PRO

4/27/2022 at 4:31 PM

Hi Mike

Thanks for the message but I am confused.

Since my last message in March you (or someone else) already approved my App for the higher app rate and this has been working successfully for a while now.

Or is this something else?

Cheers

Craig

Mike Stangel G

5/12/2022 at 2:52 PM

Craig Andrew Miles what app ID are you using?

Craig Andrew Miles PRO

5/14/2022 at 7:44 PM

Mike Stangel

I replied to you in this thread https://www.geni.com/discussions/243059 on 12/03/22 with this message:

Thanks so much for the swift reply Mike Stangel

The App ID I am using is #536

Incidentally, all of the other Apps (#532 to #535) were created "accidentally" due to my uncertainty about what was going on in the registration process although admittedly that was a while ago now.

I just tried the "Delete Application" link and it still doesnt work which was my original problem. Just gives an "Error: Page or Resource Not Found" message. The actual URL (for eg #532) is https://www.geni.com/platform/developer/apps/532 which may mean that this is a permissions error as it looks the correct URL.

Anyway I would be most grateful if you could approve App #536 and happy for you to delete #532-535 if you are so inclined. They all have the same name!

and you replied on 16/03/22

Craig Andrew MILES - done!

So I think we have already been through this but my App ID is #536 as I said and I am happy for you to delete #532-535

Mike Stangel G

5/17/2022 at 10:40 AM

Craig Andrew Miles thanks for being more on top of this than I am. :-) Turns out this is all done -- 536 is approved and 532-535 have been deleted.

Rate limit exceeded keeps failing in production

Participants:

Related Projects:

Create a free account or login to participate in this discussion