4.3.0.0: tape uploading too fragile

Found a bug in R'n'D? Report it here!

Moderators: Flumminator, Zomis

Post Reply
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

4.3.0.0: tape uploading too fragile

Post by filbo »

I finally allowed R'n'D to start uploading my tapes.

There are about 24-25000 tapes in the collection.

It uploaded some, I'm not sure how many (maybe stdout.txt & stderr.txt could be rotated through a few .1-.9 names?)

... anyway, something like 6000(?) of them. Then died with a timeout error.

I restarted and ... it's re-uploading everything from the beginning. Which means I'll need to be lucky enough to have 25000 uploads succeed with zero errors. (If it happens again, I suppose I'll be manually moving already-uploaded directories out of ~/.rocksndiamonds/tapes, and hoping it doesn't confuse the process...)

Suggestions for improvement:
  • if a single upload fails, retry it at least a couple of times
  • keep a log or something, so that next session can start from where it left off
  • when the automatically prompted initial upload session fails, serversetup.conf 'ask_for_uploading_tapes' should not be turned off
  • when any upload session fails (including things like sudden power off), the game should prompt to continue next time
  • this implies something like a commit log (starting with a flag committed to disk: 'I have an upload in progress, here are the parameters')
  • commit log doesn't need to be aggressive; this isn't a filesystem. If the upload just leaves itself a note once a minute, it'll only have to re-upload up to a minute's worth next time
  • actually, could just log upload-completed levelset names; next upload can walk the entire namespace again, quickly skipping through ones already in the log. ('complete' directories could be scanned for tape files with newer timestamps than the start of the log, to interact properly with a player who might still be adding tapes during a series of partial uploads. I'm 100% ok with using filesystem timestamps for this purpose; if someone wants to play shenanigans, there isn't really anything they can do. Marking files as 'new' makes them re-upload, but so does restarting the process. Marking them 'old' makes them not upload, which might be useful in some case, but certainly doesn't harm the server...)
  • keep stdout.txt, stderr.txt from recent runs; this could be a series of backup files .1 through .9; or append rather than overwrite, starting each run with a header in those files ('2021-11-07 01:04 -0700 Starting Rocks'n'Diamonds 4.3.0.0') (if appending, should still probably rotate the files when they get too big -- I'm thinking maybe 10MB -- but in that case only a single .old is needed for each one)
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

Re: 4.3.0.0: tape uploading too fragile

Post by filbo »

Well, this time it died at 13526 tapes uploaded. stdout.txt ends with:

Code: Select all

Tape 077:
- uploading score tape to score server - uploaded.
Tape 078:
- uploading score tape to score server - TIMEOUT.

===============================================================================
SUMMARY: Uploading tapes failed.
===============================================================================
stderr.txt ends with:

Code: Select all

[ERROR] HTTP request failed: cannot connect to host 'api.artsoft.org': Couldn't connect to remote host
(which isn't particularly informative).

... after hiding already-uploaded dirs and restarting... the first 23505 lines of stdout.txt are a litany of failing to upload one levelset after another, with a '(no tape found)' line for every single level of those levelsets. If the tape directory in question doesn't even exist, it seems like this should be a single line per levelset:

Level series identifier: 'some_thing': no tapes
Level series identifier: 'other_thing': no tapes
etc.

Then there would only be 268 lines of spurious output. Or, it could just be silent about levelsets with no tapes...
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

Re: 4.3.0.0: tape uploading too fragile

Post by filbo »

The 'tumbling hourglass' display on the R'n'D screen is ... not particularly exciting. Maybe it could have a little scrolling window of stdout below that, or a more graphical rendition of sweeping across the levels of that levelset...?
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

Re: 4.3.0.0: tape uploading too fragile

Post by filbo »

Heck, it could probably flash up a rendition of the level in question. I seem to be uploading about 2.6 tapes per second. I think RnD can render levels that quickly; not sure if it can load the level definition that quickly, though...
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

Re: 4.3.0.0: tape uploading too fragile

Post by filbo »

The next run succeeded, uploading ~10K more tapes and finishing the job. Now I've moved all my tapes back to where they belong...

So when I started last night, the server said it had around 7.5K scores. I uploaded ~24K. Today it says ~62K -- so some other player(s) must have uploaded 30K tapes over the last 12 hours!

I hope the API server will get new features, like: in the Players chart, see how many tapes belong to this player; explore their tapes by levelset and then by level; watch playback of a particular tape. In the Levelsets chart, drill down to specific levels; see whose tapes are available; watch playback. In the Levels chart, see and play tapes. In each location, every levelset, level, and player should be clickable to reach that entity's relations (levels contained in levelset, tapes recorded of level, levelsets & levels played by player)...
User avatar
Holger
Site Admin
Posts: 4073
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: 4.3.0.0: tape uploading too fragile

Post by Holger »

Just a very quick note (as it is already way to late at night for me):

I fully agree that the current upload mechanism is not robust enough, especially if you have a lot of tapes.

I have added the following now, just as you suggested (and will probably release it very soon as 4.3.0.1):

- If all existing tapes for a level set have been uploaded, the tape directory will be flagged as successfully uploaded.
- If uploading tapes fails, uploading the remaining tapes will be automatically offered at next program start.
- When uploading again, level sets that are flagged as successfully uploaded will be skipped.
- Whenever tapes for solved levels fail to upload later, the tape directory for that level set will be unflagged again.
- If a tape directory was unflagged that way, uploading missing tapes will be offered again at next program start.

In addition, I should probably also add a retry at least once after a tape upload failed, like after waiting for 30 seconds.
filbo
Posts: 647
Joined: Fri Jun 20, 2014 10:06 am

Re: 4.3.0.0: tape uploading too fragile

Post by filbo »

Thanks, those sound like good improvements. One point you seem to have missed:

- if a tape directory is flagged as uploaded, and any new tape is written into it, and the player has configured not to automatically upload tapes. the directory is unflagged

(that could also be managed in terms of filesystem timestamps -- if any tapes are newer than the directory's flag file, they're candidates for uploading. But that brings in all the potential dodginess of paying attention to filesystem timestamps...)
User avatar
Holger
Site Admin
Posts: 4073
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: 4.3.0.0: tape uploading too fragile

Post by Holger »

stderr.txt ends with:
CODE: SELECT ALL

Code: Select all

[ERROR] HTTP request failed: cannot connect to host 'api.artsoft.org': Couldn't connect to remote host
(which isn't particularly informative).
I first had the suspicion that this could be caused by the score server running out of worker processes (if requests come in faster than they could be processed), but I'm not sure about this (see below).
... after hiding already-uploaded dirs and restarting... the first 23505 lines of stdout.txt are a litany of failing to upload one levelset after another, with a '(no tape found)' line for every single level of those levelsets. If the tape directory in question doesn't even exist, it seems like this should be a single line per levelset:

Level series identifier: 'some_thing': no tapes
Level series identifier: 'other_thing': no tapes
etc.
You are right -- don't ask me why I haven't noticed this before! (Probably because I used a test installation with only a few level sets and tapes most of the time; oh well.)

Should be improved just as you described it.
The 'tumbling hourglass' display on the R'n'D screen is ... not particularly exciting. Maybe it could have a little scrolling window of stdout below that, or a more graphical rendition of sweeping across the levels of that levelset...?
Heck, it could probably flash up a rendition of the level in question. I seem to be uploading about 2.6 tapes per second. I think RnD can render levels that quickly; not sure if it can load the level definition that quickly, though...
Lots of cool stuff could be done... but I think it's not worth it for a job that runs only once for most users.
So when I started last night, the server said it had around 7.5K scores. I uploaded ~24K. Today it says ~62K -- so some other player(s) must have uploaded 30K tapes over the last 12 hours!
That's right -- user "atiE2" uploaded around 35'000 tapes in around 35 minutes in one large batch, and apparently without timeouts or other problems. That's something like 15 score tape uploads per second. Around 31'000 of these score tapes have been approved.
I hope the API server will get new features, like: in the Players chart, see how many tapes belong to this player; explore their tapes by levelset and then by level; watch playback of a particular tape. In the Levelsets chart, drill down to specific levels; see whose tapes are available; watch playback. In the Levels chart, see and play tapes. In each location, every levelset, level, and player should be clickable to reach that entity's relations (levels contained in levelset, tapes recorded of level, levelsets & levels played by player)...
Don't worry, there's more to come. :-)
Thanks, those sound like good improvements. One point you seem to have missed:

- if a tape directory is flagged as uploaded, and any new tape is written into it, and the player has configured not to automatically upload tapes. the directory is unflagged
Thanks, I indeed missed that one!
Post Reply