git.artsoft.org web site requires authentication now

[filbo]

---------- added after splitting topic ----------

To access pages on https://git.artsoft.org/ other than the start page, you have to log in with user "guest" and password "guest". Please read on for more information!

---------- added after splitting topic ----------

bug on the 25th anniversary post: where it says "the Git repository of Rocks’n’Diamonds has a special branch to compile the historic version with modern compilers and systems." it links to https://git.artsoft.org/?p=rocksndiamon ... compilable , but that gives me a 403 Forbidden error

Sorry, works again.

Blocked a bit too much in a desperate attempt to protect my web site against AI crawlers (which are totally braindead and try to fetch the same resources again and again). :(

Blocked again? Trying to open that gets a popup:

Please Sign In
https://git.artsoft.org
Username: ____
Password: ____

-- which I did not attempt to drill through. Maybe it's as easy as using our forum names & passwords? But I doubt it and did not try...

[Holger]

Blocked again? Trying to open that gets a popup:

Unfortunately, yes!

This time it's not by accident, but on purpose. I had to do it. AI scrapers were melting down my poor little server's CPU, without any chance of blocking by IP (every request comes from a different IP) or by User Agent string (it's totally spoofed), let alone by a friendly "robots.txt" (totally ignored).

But you can just log in using user "guest" and password "guest". (Totally unsure for how long this might help.)

You can also just open https://git.artsoft.org where I have added it in big text at the top of the page.

You might like to look for yourself at the last 24 hour's server monitoring what a difference this makes:

artsoft01-cpu-load.png (138.56 KiB) Viewed 126 times

I'm afraid this fight is not over yet. I hope I don't have to do the same to the forum...

[filbo]

Yeah, graph needs two different Y-axis scales for 'before' and 'after'...

This probably belongs somewhere other than the 4406 announcement thread :)

My $work is also struggling against AI scraper load. WTF is wrong with those people -- robots.txt is there for a reason (on your part); spoofed user agents and IP addresses are their for a completely nefarious reason on their part. Their 'mission' is NOT so urgent that it should intentionally blitz every host on the Internet. @*%&@#(%

[filbo]

'guest' / 'guest' is probably too obvious, and if they turn their AI even a teeny tiny bit inwards it will parse your greet text and figure out how to get past the 'firewall'. But, not sure how to do better. If you say 'login as guest with password guest-spelled-backwards' -- the AIs are certainly smart enough to figure that out also. Not sure what sort of 'Turing test' you could put into the announcement of guest-password, that humans could get past while AIs would be blocked. blah.

[Holger]

This probably belongs somewhere other than the 4406 announcement thread

Correct -- just moved it to a separate topic. (Wanted to add a new "first post", containing a quick note what it's all about, but merging posts together always sorts posts by date and time of posting, so no luck here.)

Their 'mission' is NOT so urgent that it should intentionally blitz every host on the Internet. @*%&@#(%

That's right, but there's probably too much money involved in AI crap now so as not to throw good manners overboard.

Which is quite crazy, I think. There would be nothing wrong with crawling git.artsoft.org, and Google has probably done it several times in the past without causing hiccups on the server. There's simply no reason to put such a bursty traffic load on the server, and there's especially no reason to fetch the same URLs over and over again (where "traditional" bots were clever enough to fetch the same URLs only once, even if the site has multiple links to the same URLs, as it is the case with git repos made accessbile by web using "gitweb").

'guest' / 'guest' is probably too obvious

Most probably yes.

As it is not meant to be a secret, I chose this most trivial one for the start.

and if they turn their AI even a teeny tiny bit inwards it will parse your greet text and figure out how to get past the 'firewall'.

In this case, I will have lost the battle, I think.

That is, I definitely won't try to outsmart AI scrapers that are clever enough to parse hints on the main page. It's the same with most captchas today: Machines are simply better on this than humans.

The last resort would be to post the password here in the forum, instead of a page on the site to be scraped. If that won't work anymore, I'm afraid I will simply give up.