Darkscribes Community

It is rather silly that it inter-platform federation so damn fragile. I can't wait until there's a bigger network of nodeBB sites federating.

Wild.
What's really weird is that I know I stopped receiving topics and posts from all of the Lemmy servers I follow communities on on the day I graded from 4.3 to 4.4.1, which was several days after this forum was updated to 4.4.1. Meanwhile, those communities here stopped receiving topics and posts a few days earlier than my forum did (though, I can't say for certain that that was on the day y'all updated to 4.4.1).

My follow relationships are rather mature at this point, so it's not like this is an issue with establishing new follow relationships. Old ones are broken, too. Very strange. And also, FWIW, very unidirectional: I can still post into these groups and have my content federated. I'm just not processing or receiving anything coming back my way.

I'm up to my eyeballs right now, but I'll try to look more closely into logs this weekend to see if I can identify any paradigm shifts.

New observation: I'm still receiving notifications for up/down votes from Lemmy, but no other content.

julian No, [email protected] isn't currently being followed, and it shows the warning, but all of the ones in the screen shot above are from this forum, are being followed by me, and haven't received a new topic in 8 days. [email protected], in particular, is almost hyper active right now.

After updating from 4.3 to 4.4.1, I noticed that my remote groups stopped updating. I double checked them here, and they're even further behind the group hosts, with the last received topics dating back to 16 May.

These are all Lemmy-hosted groups, so I'm not sure if it's a Lemmy-specific thing, or group actors more generally.

Huh. My Lemmy mobile client has the handle and host inverted. It doesn't look like this in the default web UI, though.

Also, my post from Lemmy hasn't landed here yet.

@[email protected] Mostly what I've observed is significant instances of timing out when trying to find communities on new instances from non-Lemmy-based websites, something that hasn't been notable from Lemmy-to-Lemmy first encounters. From the outside, it points to y'all doing some kind of compensation for possible AP issues.

I know I said "Lemmy is rough around the edges", but I really meant "ActivityPub is rough around the edges". Lemmy's just the hegemon in the AP categorized content space.

@eeeee Yeah, Lemmy is rougher at the edges than it looks sometimes. Part of the issue, I think, is that older versions of the software don't parallelize federation, so the queues can get way backed up. I've also just had follow requests from nodeBB dropped at an unusually high rate, which makes me think that Lemmy is doing things internally to compensate for some of the sharp edges of federation.

This is incredible, Julian. I'm legitimately kind of stunned by this, how well it's working already, and how quickly this got refactored.

julian So I'm been monitoring the log feed in dev mode, and noticed a couple of confusing federation hiccups, even following my re-enabling of all the default permissions on registered-users. I was hoping you could provide more insights or other areas to investigate.

Prior to changing the permissions, I was having no issues subscribing to categories from my Lemmy account, but afterwards it broke this. Reverting permissions hasn't fixed it, either. When I monitor the log, I see no evidence that the activity is reaching nodebb at all. I'm able to follow from several other Mastodon accounts, and even from some new Lemmy accounts I created on other servers.

My first thought was that Lemmy.ca has silenced me, maybe due to request rejection spam while federation was broken, but they've confirmed that I'm not on a list on their end. This has made me wonder if they might be on mine. Or if something else is still broken.

Skimming the logs, I found the following:

2025-02-26T17:28:46.883Z [4566/160807] - verbose: [activitypub/verify] Starting signature verification...
2025-02-26T17:28:46.883Z [4566/160807] - verbose: [activitypub/verify] Retrieving pubkey for https://lemmy.ca/u/Kichae#main-key
2025-02-26T17:28:46.887Z [4566/160807] - verbose: [activitypub/get] https://lemmy.ca/u/Kichae#main-key
2025-02-26T17:28:47.116Z [4566/160807] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] Request body check passed.
2025-02-26T17:28:47.125Z [4566/160807] - verbose: [middleware/activitypub] Origin check passed.
2025-02-26T17:28:47.131Z [4566/160807] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-26T17:28:47.134Z [4566/160807] - verbose: [activitypub/inbox/undo] Like https://sopuli.xyz/post/22661764 via https://lemmy.ca/u/Kichae
[api] Exception caught, error with stack trace follows:
Error: [[error:invalid-pid]]
    at inbox.undo (/var/www/html/nodebb_wap/src/activitypub/inbox.js:521:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

pid I'm assuming is post ID? I might be reading this totally wrong, but it looks like it's trying to process an upvote I made on this post, but doesn't know where to find it. This is a post that I'd previously experimented with moving from World into a category, on a previous iteration of the forum. I've purged the database and totally reinstalled since then, and the post is not currently located on the forum anywhere -- it has not been re-imported, and currently cannot be due to the communication issues.

I'm also receiving a lot of activities in the log that look like this, that I suspect may be related, but do not yet have the skills to interpret:

2025-02-26T18:28:50.637Z [4566/160807] - verbose: [activitypub/send] https://lemmy.world/inbox
2025-02-26T18:28:50.674Z [4566/160807] - verbose: [activitypub/send] Could not send Reject to https://lemmy.world/inbox; error: {"error":"unknown","message":"Failed to parse object https://wanderingadventure.party/actor with content {\n    \"@context\": [\n        \"https://www.w3.org/ns/activitystreams\",\n        \"https://w3id.org/security/v1\"\n    ],\n    \"id\": \"https://wanderingadventure.party/actor\",\n    \"url\": \"https://wanderingadventure.party/actor\",\n    \"inbox\": \"https://wanderingadventure.party/inbox\",\n    \"outbox\": \"https://wanderingadventure.party/outbox\",\n    \"type\": \"Application\",\n    \"name\": \"Wandering Adventure Party\",\n    \"preferredUsername\": \"wanderingadventure.party\",\n    \"publicKey\": {\n        \"id\": \"https://wanderingadventure.party/actor#key\",\n        \"owner\": \"https://wanderingadventure.party/actor\",\n        \"publicKeyPem\": \"-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1iTGf1EX34ox09yxpS44\\nvUdZdkle3fXlQbb4Q6kDGFcm8VMI1iv5DYPBt0CQXZ7vIDfLq0P4qKN810H1THiB\\ndkrcQ5agxXWXsPwMuCfyC7umE89/tCUpEr7+ghjOMWreHQb7MeIvrYz0WEK2Yiva\\nE3L8pkxvwyKXA2pvU10fewsGAjiCSAj30IZ6EoaDuUK9OlRiKku6U4wI6qdun90q\\nl3K/aSYAhI7NOAgXB6RsjEg0TFgSt1DFQDWIZcDpdG8iGpysyNQwUmc2gaTNWtKy\\nuDkILJO6HeVquku8o+/cB+dXh5G3cqVx5J08bH3raSyo1BOTynHX3tvblgP2+jPH\\npQIDAQAB\\n-----END PUBLIC KEY-----\\n\"\n    }\n}: data did not match any variant of untagged enum PersonOrGroup"}
2025-02-26T18:28:50.674Z [4566/160807] - verbose: [activitypub/send] Added Reject to https://lemmy.world/inbox to retry queue for 1024000ms

I receive a steady stream of them from lemmy.world and sopuli.xyz, both of which I'm having federation issues with. But there's nothing like this from lemmy.ca, so I might be barking up the wrong tree.

Do you have any thoughts?

Yup. Things are flowing as normal once more.

That was my first assumption, so I've already made sure that 'fediverse' has full privileges on Uncategorized.

More digging seems to have surfaced registered-users as the culprit. I had stripped the group back to just viewing privileges, and was playing with using additional groups and rewards to grant expanding posting rights, but the fediverse group privileges didn't seem to override these.

Is that by design?

Finally got around to actioning this. Pinging myself from activitypub.academy got a 403 response: "Sending failed (htps://wanderingadventure.party/inbox responded with status 403 Forbidden)", and generated the following in the console log while in dev mode:

2025-02-24T15:58:18.922Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T15:58:18.923Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T15:58:18.927Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.117Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.370Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T15:58:19.377Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Asserting 1 actor(s)
2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Processing https://activitypub.academy/users/braulus_aelamun
2025-02-24T15:58:19.383Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun
2025-02-24T15:58:19.390Z [4566/131736] - warn: Route requested but not found: /api/v1/timelines/public?limit=40
2025-02-24T15:58:19.415Z [4566/131736] - warn: Route requested but not found: /api/v1/streaming/public
2025-02-24T15:58:19.799Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/followers
2025-02-24T15:58:19.802Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/following
2025-02-24T15:58:20.310Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T15:58:20.317Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517
2025-02-24T15:58:20.777Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517 contains no context.
2025-02-24T15:58:20.819Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found.
[api] Exception caught, error with stack trace follows:
Error: [[error:no-privileges]]
    at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Promise.all (index 0)
    at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3)
    at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

No such errors occur, however, if I follow the user account:

2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:21:10.504Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:21:10.958Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:21:10.959Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:21:10.971Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:21:10.972Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)...
2025-02-24T16:21:10.978Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved.
2025-02-24T16:21:11.014Z [4566/131736] - verbose: [activitypub/send] https://activitypub.academy/inbox
2025-02-24T16:21:11.466Z [4566/131736] - verbose: [activitypub/send] Successfully sent Accept to https://activitypub.academy/inbox

Nor if I send a direct message:

2025-02-24T16:25:27.790Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:25:27.792Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:25:27.798Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:25:27.802Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.

Following the activitypub.academy account also seems to work fine:

2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:27:54.808Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:27:54.811Z [4566/131736] - verbose: [activitypub/inbox.delete] Object (https://mastodon.social/users/goma0) does not exist locally. Doing nothing.
2025-02-24T16:27:55.216Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:27:55.217Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check failed, stripping object down to id.
2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)...
2025-02-24T16:27:55.237Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved.

But when I post publicly from it, the messages bounce from my nodebb site, once again with a 403 error:

2025-02-24T16:30:11.720Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:30:11.728Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:30:11.732Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:30:11.741Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026
2025-02-24T16:30:12.190Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026 contains no context.
2025-02-24T16:30:12.203Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found.
[api] Exception caught, error with stack trace follows:
Error: [[error:no-privileges]]
    at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Promise.all (index 0)
    at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3)
    at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

While skimming the output stream, I'm also noticing a lot of entries from Mastodon servers reporting 410 errors:

2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.874Z [4566/131736] - verbose: [activitypub/get] https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Received 410 when querying https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Error received: Gone
2025-02-24T16:24:36.889Z [4566/131736] - verbose: [activitypub/verify]   Failed, key retrieval or verification failure.

It's not clear to me what these are, though, as when I try to boost content or post from my main Mastodon account, which is being followed by my nodebb account, I get the 403 error reported a the top of this post.

Is there some kind of conflict with respect to file ownership that's causing this? Or is this about the fediverse pseudo-user's permissions to access different categories? Because the linux user running nodebb should have full ownership rights to the folder, but it's possible I've broken that somehow. Meanwhile, I've absolutely been playing with category access rights, but I've reset them all since this started.

Sometime yesterday my forum stopped receivng (or accepting?) new remote topics or posts. I'd spent time yesterday playing with plugins and permissions, so I've been rolling back those changes this morning, but it hasn't resulted in the resumption of content flow.

The only thing the logs indicate is that the forum was receiving dislikes and didn't know how to handle them (at some point, I denied the federated user group the ability to downvote), but last night it just stopped complaining about this. I assumed at this point maybe I'd been spamming some kind of error at the other servers and got blocked or something, but I can still chat (two way) with my remote accounts, and remote accounts can see my posts on the forum. I don't receive any replies, though, weirdly enough, I do receive likes/upvotes.

Are there other logs I should be looking at to make sense of this?