Different behaviour between sync and gthread workers with preload=True

[mihalikv]

Gunicorn version: 23.0.0 also 20.1.0

Problem:

I have application in Django 4.2, we are running this application in production with following settings:

• workers = cpu.count + 1
• threads = 8
• preload=True

When I tried to change these configuration my best result(RPS, request duration) was with these settings:

• workers = cpu.count + 1
• threads = 1
• preload = False

Hovewer I would like to use preload=True but problem is that application start crashing when sync workers are user together with preload=True.

Settings	Does it work ?
workers=8 threads=1 preload=False	Yes
workers=8 threads=2 preload=False	Yes
workers=8 threads=2 preload=True	Yes (something like this is currenly used in production)
workers=8 threads=1 preload=True	No

Exception:

Basically exception are different, but generally there is something wrong with DB/cache and memory sharing, I guess.

Expectations:

I understand that my app may have some flaws, and preload might be sharing memory incorrectly, which causes these exceptions.
However, I don't understand why this is not a problem when gthreads are used?

[pajod]

Gunicorn isnt doing much here, other than importing your not-thread-safe object in a multithreaded setting, and it is unlikely that any change in Gunicorn can diagnose or fix your problem.

You probably want to file a bug report with cacheops, providing them with a minimal reproducer of what query exactly (hint: could be earlier than your traceback, happening in migrations already!) issued by django cms should not have resulted in multiple objects returned. You may be able to prove that this is a cache invalidation issue by making that query yourself manually from your app, and thus determining whether you are receiving duplicate or stale objects.

[mihalikv]

Thanks for your answer.
I'm just curious why it works with threads=2. It should fail in the same way. So I'm thinking that it's actually not a problem in the app itself. However, I'm currently not able to create a minimal reproducible example.

[pajod]

Oh, you think this is not about thread-safety later on after all, but actually caused in the sync worker only? Initiating and keeping a database connection at import time (before fork()) comes to mind. That should be easy to prove: ask Django to dump its django.db.connections on some URL unrelated to crm (some URL that works even when the database does not).