Increased memory consumption after upgrading from v3.7.x to v4.2

[lassesuom@community.nodebb.org]

Hey, so after the upgrade the memory consumption has been creeping up at a fairly steady rate and exceeding 1GB of memory. Is this something to be expected from the version upgrade? Or could it be due to our own changes / plugins we have? Or due to scrapers perhaps?

Upgrade happened on Monday 14th and before that the memory consumption was fairly steady at around 500MB. After the upgrade the memory consumption has been steadily going up during the day as seen in the pictures:

Screenshot when the sudden increase was noticed:

Here is last 7 days: (the falling edges are due to daily restart)

Screenshot from the /admin/development/info on the 23rd when it started also using swap.

Active plugins:
* [email protected] (installed, enabled)
* [email protected] (installed, disabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, disabled)
* [email protected] (installed, disabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, disabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, disabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)
* [email protected] (installed, disabled)
* [email protected] (installed, disabled)
* [email protected] (installed, disabled)
* [email protected] (installed, disabled)
* [email protected] (installed, enabled)
* [email protected] (installed, enabled)

Nginx is being used to serve static assets and the config is the same as it was before the upgrade.

Setup:
Mongodb Atlas M40
Redis is Heroku's own key value store premium 2
Heroku -> 4 dynos (Standard-2X variant)
In one dyno we have nginx + nodebb

Any help would be greatly appreciated!
Thanks in advance.

[baris@community.nodebb.org]

I would disable the daily restarts to see if the memory usage keeps going up or stabilizes.

[lassesuom@community.nodebb.org]

I can't disable it. The auto restart is a feature from our hosting provider

[julian@community.nodebb.org]

Are you using the ActivityPub functionality? It might be related to that if only because that's what's new (among other smaller items) in v4.

If you toggle the global activitypub switch off, does that make memory usage stabilize?

Keep in mind that since you upgraded, ActivityPub is automatically disabled, so perhaps this is unrelated.

[lassesuom@community.nodebb.org]

ActivityPub hasn't been touched at all so it has been disabled the whole time.

[lassesuom@community.nodebb.org]

On a second note, is it normal behavior that redis has gotten 440k new keys in 4 days ? In last 7 days there has been less than 100 new topics created.. What kind of data is stored there? I guess sessions but what else?

[baris@community.nodebb.org]

Are you using redis only or mongodb + redis? If redis only all forum data and sessions are stored there.

You can use redis insight to see keys. Click the tree view to group them by namespace.

[julian@community.nodebb.org]

@lassesuom my guess is they're sess:* keys. They can balloon up if you have lots of guests *cough* AI scrapers.

There used to be logic so that guests didn't receive a session key (since they didn't really need one for regular browsing) but it could be that that detection no longer works.