I'm noticing a huge surge in 404 errors on my website because a bunch of fediverse crawlers seem to think that beesbuzz.biz is a Mastodon instance. I wonder what gave them that idea.

It's not a huge deal or anything, but it's also a bit baffling why these crawlers have each been retrying every 10 minutes for the past few days. fediverse.space and fediverse.network are the two big ones but there's a handful of others as well.

@fluffy fediverse.network tries all the possible URLs

if it's happening every 10 minutes you've enabled monitoring

(log in and disable monitoring to make it slow down)

@ben How would I have enabled monitoring and how do I log in? I don't have an account there or anything, I've never even heard of this before just now.

@fluffy actually, it looks like fediverse.network doesn't have monitoring enabled for that domain

fediverse.network/beesbuzz.biz

so the "every 10 minutes" thing is probably fediverse.space (which I've never heard of)

@ben okay, you're right, fediverse.space is doing it every 10 minutes. fediverse.network is doing it every 30 minutes. Which is still a lot more often than it should for something that is not and never will be a Mastodon instance.

@fluffy yeah, @href is aware the crawler needs better detection of non-instances

@ben @href I'd think returning a 404 on /api/v1/instance would be a pretty good indication :)

@fluffy @href yeah but the crawler isn't very smart (yet)

@ben @href Any idea why it would have started doing this just today?

@fluffy hmm, maybe some instance claims it's a peer? are there any activitystreams documents on there?

Follow

@ben I've been using Bridgy Fed to republish my stuff as @beesbuzz.biz for a while but that hasn't changed at all since last October. Unless something changed on Bridgy Fed itself today. @snarfed.org ?

nope, no recent bridgy fed changes. it definitely does make the fediverse think your site is an activitypub/mastodon instance, which will trigger requests like these, but that’s always been true. more details: https://github.com/snarfed/bridgy-fed/issues/38

@fluffy @ben @beesbuzz.biz nope, no recent bridgy fed changes. it definitely does make the fediverse think your site is an activitypub/mastodon instance, which will trigger requests like these, but that’s always been true. more details: github.com/snarfed/bridgy-fed/

@snarfed @ben @beesbuzz.biz I wonder what caused things to change such that I suddenly was getting the influx of traffic then.

@snarfed actually it looks like this has been going on for a while, it just only happens every few days and lasts for the whole day and then they stop for a while. Whatever.

@fluffy @snarfed probably what happened is some instance claimed your domain as a peer and the two spiders picked that up at around the same time

@ben @snarfed here's my stats graph from the last couple of weeks. each of those 404 spikes is one of the instance lists going weird, and then it stops for a while.

I suspect that what's happening is some instance where someone's following me via bridgy fed gets set off whenever I post a new blog entry or something.

@fluffy @snarfed fediverse.space just launched today so it might have been in development during that time

@ben @snarfed I’m tempted to just add a Mastodon instance info route to my site just to make it a means of advertising publ :P

Sign in to participate in the conversation
Queer Party!

A silly instance of Mastodon for queer folk and non-queer folk alike. Let's be friends!