Mistral AI Rate limiter

[monochrome]

I have problems getting Mistral up to my rate limit of 6 requests per second. I've set the request-per-minute limit to 360 in the settings, and the token limits are all set to the appropriate values. 

It's hovering around something like 0.5 rps, and the log file is full of "Request rate limit expired for [700] pixtral-12b-latest. Was 1899-12-30T00:00:00.0"

There's also this in the log, which I interpret as having lots of remaining usage quota:

Code Select Expand

ratelimitbysize-limit:2000000
ratelimitbysize-query-cost:173
ratelimitbysize-remaining:1995890
ratelimitbysize-reset:55
x-ratelimitbysize-limit-minute:2000000
x-ratelimitbysize-limit-month:10000000000
x-ratelimitbysize-remaining-minute:1995890
x-ratelimitbysize-remaining-month:9999391322

[Mario]

Do you have a paid account? Where did you read the six requests per second?

IMatch limits Mistral to one thread and 1 request per second, based on the information I have read on their web site at the time I've implemented Mistral support.

[monochrome]

It's a paid personal account. This is what's in the Mistral console. I plugged in the numbers into the IMatch rate limiter. (For the record, one image is about 1000 tokens, give or take, so I'm pretty much limited by the rps.)

[Mario]

I don't follow.

Mistral differentiates between LeChat (similar to paid ChatGPT) and 
API usage (Free/Experimental plan with 1 rps) or "Pay as you go". I have tested this with the Experimental option, not with a paid plan so far.

Where did you create a personal account for the API? From which page did you copy these numbers?

For pay as you go, I get these limits:

[monochrome]

This is from the "pay as you go" limits page - https://console.mistral.ai/limits/ . I have the "Scale" subscription just as you posted.

[Mario]

A scale it is, that I can understand.

I've contacted their support to see if I also need to enter a subscription to try out higher RPM settings for some load tests.
I've already added the Pixtral-Large model (the good and expensive one) for the next release.

I have 10€ in credits, but these seem to apply only to LeChat, not to the API. With OpenAI I liked the fact that I can add 10$ pre-paid and then use it up. Stil have 4$ from my payment in June 2024.

With Mistral it seems to be the monthly usage limit in € that protects from spending too much money on their platform. I will do that in order to test if I can losen the thread count and parallel threads for Mistral. The support "usually" responds within 3 hours, so I will know more than.

[Mario]

Can you try this:

Make a backup copy of the file

c:\ProgramData\photools.com\IMatch6\config\ai-services-202501.json

and then open the original in a text editor (Notepad, VS Code).
Search for pixtral-12b-latest
Scroll down until you see "maxThreads": 1
Change this to "maxThreads": 8

IMatch limits the number of parallel AutoTagger threads to this maximum.
It also limits the number of parallel threads to 1 when the request per minute rate limit is < 120.
Set the RPM to 300 for example.

I've switched to the pay as you go plan and tested this successfully with about 2,000 images.
It's very fast.

[monochrome]

Set the RPM to 300 for example.

I set it to 360, since I have 6 rps.

Change this to "maxThreads": 8

Oh yeah, that hit the spot!

If the estimate is correct, it's going to be 24k photos in 2h or 3 images/second. This is awesome.

[Mario]

Sounds good. A limit of 6 requests per second of course does not mean that the AI responds 6 times per second. At, say, 2 or 3 second per response, this will average out at the processing times you see.