Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

R'n'D is always evolving. Check here to see if a new version is out.

Moderators: Flumminator, Zomis

Post Reply
User avatar
Holger
Site Admin
Posts: 3633
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by Holger »

Release candidate 5 of R'n'D 4.3.0.0 is available now:

https://www.artsoft.org/RELEASES/linux/ ... rc5.tar.gz (Linux)
https://www.artsoft.org/RELEASES/mac/ro ... .0-rc5.dmg (Mac)
https://www.artsoft.org/RELEASES/window ... -win32.zip (Windows / 32 bit)
https://www.artsoft.org/RELEASES/window ... -win64.zip (Windows / 64 bit)
https://www.artsoft.org/RELEASES/androi ... .0-rc5.apk (Android)

Or in a web browser:

https://staging.artsoft.org/rocksndiamonds/play

Main changes since RC4:

- fixed leaking open sockets when doing HTTP requests

This is a major bugfix version and especially important for all people who have around 1000 tapes or more, which resulted in serious problems in the previous versions when trying to upload those tapes to the high score server.

Please let me know of any problems or findings you might encounter. (I am a bit careful now if this version could be the official release version or not. We will see... :-) )
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

This is a major bugfix version and especially important for all people who have around 1000 tapes or more, which resulted in serious problems in the previous versions when trying to upload those tapes to the high score server.
I think this fixed the major 'serious' problem. Lots more tapes submitted this time around, way closer to the expected count. :) And the levels are still there after submitting tapes! :)

Code: Select all

/usr/bin/find ~/.rocksndiamonds/tapes -type f -name '[0-9][0-9][0-9].tape' | /usr/bin/wc -l
Comparing 'SUMMARY tapes uploaded' to this implies that there may be at least 61 tapes that didn't get submitted that could be submitted. (way better than having around 8k nonsubmitable tapes)
Could these be ?random tapes in random folders? ?backups? ?downloads from others?
Don't know how to find out which ones they were.
[UPDATE: in there found an extra 'EEM 1' with 23 tapes that came from a zip that's also in there. The stderr shows a possible ~14 error tapes?, assuming those didn't add to upload count, this still leaves 24 unaccounted for?]
I also saved stdout and console output for this run if needed.
Thanks!
Attachments
stderr.txt
(1.52 KiB) Downloaded 16 times
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

Feature request:
Under 'SETUP' > 'GAME AND MENU' > ...
There's a
'ONLY SHOW LOCAL SCORES'
How about splitting this up in to something like:
'SHOW LOCAL SCORES' in 'HALL OF FAME'
and
'SHOW REMOTE SCORES' in 'HALL OF FAME'
?
This way people can verify which, if any, of their scores got accepted into high score server. (by not showing 'LOCAL', and only showing 'REMOTE')
filbo
Posts: 566
Joined: Fri Jun 20, 2014 10:06 am

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by filbo »

The existing single option covers it, as long as server-known scores are marked somehow in the display. e.g. a different color, or a mark in a new column...?
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

filbo wrote: Thu Oct 21, 2021 6:12 am The existing single option covers it
There's no way 1 bit to cover 4 possibilities. Currently it does 'local+remote' OR 'local only'. No option for 'remote only', nor 'neither remote nor local'(an essentially useless option of the four).
filbo wrote: Thu Oct 21, 2021 6:12 amas long as server-known scores are marked somehow in the display. e.g. a different color, or a mark in a new column...?
This could kind of make up for no 'remote only' option. Without 'remote only' option there may be several local scores to have to also look through when trying to find which one of those, if any, is shown 'remote', and also as a way to find out if a 'tape' is bad. (There should be an easier way to list which tapes were declared bad)
User avatar
Holger
Site Admin
Posts: 3633
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by Holger »

I think this fixed the major 'serious' problem. Lots more tapes submitted this time around, way closer to the expected count.  And the levels are still there after submitting tapes! 
I'm really happy to hear that it works now! :-)
Could these be ?random tapes in random folders? ?backups? ?downloads from others?
Don't know how to find out which ones they were.
Just have a look at "stdout.txt" and compare it with your "levels" and "tapes" folders (probably no need to compare single levels/tapes, just level sets). If you don't have it anymore (you only attached "stderr.txt"), just re-create it by submitting all tapes again -- it does not hurt, as it's only a test server. :-)

I suspect that you have some tapes for a level set that is not in your "levels" folder anymore, or with a slightly different name, or something like that...
How about splitting this up in to something like:
'SHOW LOCAL SCORES' in 'HALL OF FAME'
and
'SHOW REMOTE SCORES' in 'HALL OF FAME'
?
Very good point. The current "only show local scores" option was an attempt to make it possible to show scores just like before, without showing the new server scores (as disabling using the score server does not delete cached scores already downloaded from the server).

But you are right, the current option is incomplete somehow.

The disadvantage of your suggestion is that it would be possible to disable both options, which would result in a Hall of Fame being always empty, which would be a bit silly. :-)

I think I should add a selectbox style option with the three options "local scores only", "remote scores only" and "local and remote scores", so you always have a sane state.
Without 'remote only' option there may be several local scores to have to also look through when trying to find which one of those, if any, is shown 'remote', and also as a way to find out if a 'tape' is bad. (There should be an easier way to list which tapes were declared bad)
I think I did not get the point here; maybe some more explanation?
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

Holger wrote: Thu Oct 21, 2021 6:57 pm
Could these be ?random tapes in random folders? ?backups? ?downloads from others?
Don't know how to find out which ones they were.
Just have a look at "stdout.txt" and compare it with your "levels" and "tapes" folders (probably no need to compare single levels/tapes, just level sets). If you don't have it anymore (you only attached "stderr.txt"), just re-create it by submitting all tapes again -- it does not hurt, as it's only a test server. :-)

I suspect that you have some tapes for a level set that is not in your "levels" folder anymore, or with a slightly different name, or something like that...
I think I may have found them...

Code: Select all

#!/bin/sh
cd ~/.rocksndiamonds/tapes || exit
for x in * ; do if [ "$(/bin/grep -c "$x" -- ~/.rocksndiamonds/stdout_4.3.0.0rc5.txt)" -eq 0 ] ; then /bin/echo "$x" ; /bin/ls "$x" ; fi ; done
Time to resubmit and see if the counts match ....
Seemed to be working for a while, then this...
CANNOT UPLOAD
TAPES TO SCORE
SERVER!
YOU CAN UPLOAD
YOUR TAPES FROM
THE SETUP
MENU
LATER!
- uploading score tape to score server - TIMEOUT.

===============================================================================
SUMMARY: Uploading tapes failed.
===============================================================================
I still have internet access. Last time I did this it took way too long. All tapes are less than 17MB in total. Even at a slow 1Mb/s that would be only be 2m16s, but this takes several times longer.(when it works)
When playing earlier I noticed a few of these message in the console.
[ERROR] server failed to handle request: 500 Internal Server Error
I suppose these were from winning a level.
Will this try to auto submit these levels again?
Holger wrote: Thu Oct 21, 2021 6:57 pm
How about splitting this up in to something like:
'SHOW LOCAL SCORES' in 'HALL OF FAME'
and
'SHOW REMOTE SCORES' in 'HALL OF FAME'
?
Very good point. The current "only show local scores" option was an attempt to make it possible to show scores just like before, without showing the new server scores (as disabling using the score server does not delete cached scores already downloaded from the server).

But you are right, the current option is incomplete somehow.

The disadvantage of your suggestion is that it would be possible to disable both options, which would result in a Hall of Fame being always empty, which would be a bit silly. :-)

I think I should add a selectbox style option with the three options "local scores only", "remote scores only" and "local and remote scores", so you always have a sane state.
Agree
User avatar
Holger
Site Admin
Posts: 3633
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by Holger »

- uploading score tape to score server - TIMEOUT.
No idea what might be the reason for this timeout, and I also cannot see any errors in the server log file. (You have successfully uploaded some ~3000 tapes to the server during that batch run, and all are marked as "OK" in the server logfile from your IP address.)
I still have internet access. Last time I did this it took way too long. All tapes are less than 17MB in total. Even at a slow 1Mb/s that would be only be 2m16s, but this takes several times longer.(when it works)
The point is that this is not one single connection, transferring all ~9000 tapes (which may be 17 MB) in one data transmission, but there is a single connection opened for each tape, tape content (usually only a few KB) is transferred, connection is closed, next connection is opened for next tape ... and so on. This causes a huge overhead, so it will take considerably longer than those calculated 2m 16s.

It would be possible to extend both the client and the server software to support transferring all tapes in one connection, but I don't see a point in spending that effort that is only used once for each player (and never for new players).

But yes, you need a stable internet connection during tape upload, as one single timed out connection (for whatever reason) stops uploading all remaining tapes. In this case, you can try again from the setup menu, and hope for a better network connection next time.

Sorry that I cannot offer a better, more stable solution with my limited programming resources. :?
When playing earlier I noticed a few of these message in the console.
[ERROR] server failed to handle request: 500 Internal Server Error
I suppose these were from winning a level.
Yes, exactly -- but not from submitting the score (which still worked fine), but from getting all other scores for the Hall of Fame from the server.

This indeed was a programming error I introduced last night when doing a change required for mapping non-sanitized level set identifiers (like "Walpurgis World") to canonical, sanitized level set identifiers (like "Walpurgis_World"), which is required for verifying score tapes.

This bug is fixed now, and you should not get this error again.
Will this try to auto submit these levels again?
R'n'D does not automatically try to submit your score tapes again, but you have to start it again from the setup menu -- just like the message dialog quoted above said. :-)

So please try again uploading your tapes -- I assume that it was just a temporary network hiccup (like getting a new IP address from your provider, which some providers do every 24 hours, at least here in Germany). In general, this should work fine (even if it takes longer than expected), and the last time you tried it, it apparently worked fine (uploading something around 8000 or 9000 tapes then).
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

Holger wrote: Fri Oct 22, 2021 10:29 am It would be possible to extend both the client and the server software to support transferring all tapes in one connection, but I don't see a point in spending that effort that is only used once for each player (and never for new players).
Will this try to auto submit these levels again?
R'n'D does not automatically try to submit your score tapes again, but you have to start it again from the setup menu -- just like the message dialog quoted above said.
Would something like this speed things up...
Compress to a file locally, Extract file on server
find -newer oldZipFile # or find stored date of zip creation start
or even better
You can do this directly with tar and even better:
tar -N '2014-02-01 18:00:00' -jcvf archive.tar.bz2 files
This instructs tar to compress files newer than 1st of January 2014, 18:00:00.
source: https://stackoverflow.com/questions/848 ... after-date
Of course this assumes trust on the system time, tar, etc...
-
After removing stuff found by code in earlier post,
Time to resubmit and see if the counts match, try 2....
Now R'n'D submitted count is only down by 1. Is R'n'D counting the 1st item as 'zero', and `find` counting 1st item as 'one''?
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

This indeed was a programming error I introduced last night when doing a change required for mapping non-sanitized level set identifiers (like "Walpurgis World") to canonical, sanitized level set identifiers (like "Walpurgis_World"), which is required for verifying score tapes.
Would a user replacing 'spaces' with '_' in folder names for the ~/.rocksndiamonds folders('levels', 'levelsetup', 'scores', 'tapes', ...) cause any problems?

Code: Select all

find ~/.rocksndiamonds -type d -name '* *' -exec printf "mv \"{}\" " \; -exec sh -c 'echo \"$1\" | tr " " "_" ' sh {} \;
note: This code only show how things could be changed, it doesn't change anything.
Last edited by BryanFRitt on Fri Oct 22, 2021 11:57 pm, edited 2 times in total.
User avatar
Holger
Site Admin
Posts: 3633
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by Holger »

Would something like this speed things up...
Compress to a file locally, Extract file on server
Compressing all tapes to a zip file would most probably make uploading them faster than uploading them uncompressed. But the point is that all this would require a lot of extra programming that is only needed once, and then never again (and never at all for new users). So I see this as a waste of my precious time. ;-) Just start the upload, don't hold your breath while it's uploading your tapes, and if it takes too long to wait for it, use it for a coffee break. :-)

BTW: It seems your upload was successful. There are 9,526 tapes uploaded today, starting at 18:24:53 UTC.
Now R'n'D submitted count is only down by 1. Is R'n'D counting the 1st item as 'zero', and `find` counting 1st item as 'one''?
R'n'D starts a counter with 0 and increments it for each uploaded tape. So if you upload 10 tapes, it will then report 10 uploaded tapes. :-)
Would a user replacing 'spaces' with '_' in folder names for either the 'levels' folder and/or 'tapes' folder and/or 'scores' folder and/or 'levelsetup' folder, ... cause any problems?
No, that's fine. Just make sure that you don't forget any of those directories, or R'n'D cannot find the corresponding tapes, or scores, or ...

But you don't have to do this, as long as you do not plan to auto-validate tapes on the command line using the "-e" switch and the "autotest" command (or generally the "autoplay" family of commands), as it treats spaces as separators for level set identifier and level numbers when parsing strings like "This_cool_level_set 001 002 003". So renaming a level set from "This cool level set" to "This_cool_level_set" does not hurt on the server, and may be required when using "autotest" locally. However, renaming it to "This_Cool_Level_Set" would result in a new level set to be "detected" (which will then be seen as missing on the score server, as there is only "This_cool_level_set" -- the file system used on the server is case sensitive).

But maybe I should transform all level sets on the score server to lowercase letters, or something like that, as people might like to adjust level set directories to use either upper or lower case letters on their computer...
BryanFRitt
Posts: 126
Joined: Mon Nov 13, 2017 4:16 pm

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by BryanFRitt »

Holger wrote: Fri Oct 22, 2021 11:56 pm only needed once, and then never again (and never at all for new users)
Will this try to auto submit these levels again?
R'n'D does not automatically try to submit your score tapes again, but you have to start it again from the setup menu -- just like the message dialog quoted above said.
and what if I user completes levels offline? Scores would be forever only local only, unless they wait for the entire tape collection to upload again.

some ideas to shorten things down...

<a> option by_datetime)
A submit tapes after a DATETIME would be really helpful. Store start of last successful SUBMITTED_ALL_BEFORE_DATETIME datetime, and use that as a datetime to submit tapes after, if the update was successful, update SUBMITTED_ALL_BEFORE_DATETIME. This datetime could also be updated on every successful upload so long as there hasn't been a upload failure.
[for find] Instead, you can use -newermt '2009-01-03T12:31:43' to avoid the need to create a file just for reference. –
Michael Mior
Jan 17 '12 at 17:43
source: https://stackoverflow.com/questions/848 ... after-date

<b>option submission_status_files)
I guess this could also be done with a 'submitted' flag for each tape. When a tape gets created or modified set it's 'submitted' flag to 'false'. Upon successful upload set it to 'true'. When going through tapes to submit, ignore all the ones that have 'submitted' as 'true', if 'submitted' is 'false' try to submit the tape. Then set it's submitted to 'true' if the submission was successful.

<c>option not_submitted_flag_files)
Actually, I suppose after a successful initial submit everything, it could just assume submitted is 'true', and only submit the tapes that have flags to submit them that are created after a successful run that hasn't gotten submitted. When a level gets completed, create a note file to say this tape could be submitted. Delete the note file when the tape gets successfully uploaded, or the tape gets updated to run that doesn't end successfully.

There could be a submit everything, and submit only tapes flagged to be submitted. My current preference seems to be <c> and <a>. Can something like <a> be done without even modifying R'n'D code? Can one just submit an individual level/tape without replaying it?

Either way these sound like they might could be done without any changes to server side of things, only local R'n'D code changes. (If you hate server coding) And could save the server from a bunch of unnecessary processing.
User avatar
Holger
Site Admin
Posts: 3633
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: Rocks'n'Diamonds 4.3.0.0 Release Candidate 5 released!

Post by Holger »

Thanks a lot for your suggestions, they really are a good source of inspiration! And I really like the ideas you had regarding upload of tapes that haven't been uploaded yet, and also regarding the problem of partially uploaded tapes! Especially regarding problems like this: If somebody has a huge number of tapes, say, something like 10'000 tapes (like in your case), but has a very slow and/or non-stable network connection, so tape upload regularly fails at around 5'000 tapes, even retrying upload would not help, as the upload process fails again after some time. Any kind of book keeping (to mark which tapes have already been uploaded) would solve this problem -- you could even upload tapes in 10 runs, always uploading a few more tapes, until all tapes have been uploaded. I really like your ideas that would make this possible, and most of them require only minor coding on the client side.
and what if I user completes levels offline? Scores would be forever only local only, unless they wait for the entire tape collection to upload again.
That's right indeed -- and currently they would either be lost forever (for the server), or you would have to manually trigger a complete re-upload (which most people just won't do, as they are not able to do the required editing of setup files).
<a> option by_datetime)
This would be a very good way to upload only newer tapes, and easy to implement.

The only disadvantage I can see is that this could not be used to complete an all-tapes upload that failed halfway.
<b>option submission_status_files)
...
<c>option not_submitted_flag_files)
Both ideas look fine for its purpose; I prefer variant (c) as it does not require modifying the tape itself, but uses a separate file to flag the "tape can be submitted" state. (In fact, it's a cool idea not to use a "this tape has been submitted" file, but a "please submit this tape" file, as it does not waste file storage after upload.)
Can something like <a> be done without even modifying R'n'D code?
Definitely yes! You can initially start with "upload *all* tapes", and upload only newer files later. What you can *not* do that way would be to continue a failed "all tapes" upload.
Can one just submit an individual level/tape without replaying it?
Definitely yes, again -- it's as easy as the following commands:

Code: Select all

# Upload all my tapes for a given level set:
$ ./rocksndiamonds --mytapes -e "autoupload rnd_tutorial_niko_boehm"

# Upload a single tape for a given level set:
$ ./rocksndiamonds --mytapes -e "autoupload rnd_tutorial_niko_boehm 001"

# Upload several tapes for a given level set:
$ ./rocksndiamonds --mytapes -e "autoupload rnd_tutorial_niko_boehm 001 002 003"
Just noticed that these commands are not yet listed with the "--help" option! :shock:
Either way these sound like they might could be done without any changes to server side of things, only local R'n'D code changes. (If you hate server coding)
Definitely yes, again! No server changes required for the above ideas! (And no, I do not hate server coding, quite the opposite: It's very nice to be able to do changes that do not require rolling out a new client version, and then... writing server side code in PHP is so much faster and easier than writing client code in good old C. You can use whatever library you want, without making sure that it is available (and works) with Linux, Mac, Windows, Android... It's a lot more fun, and *much* faster to get things done. :-) )
And could save the server from a bunch of unnecessary processing.
Although it does not hurt that much (my servers are fast and powerful ;-) ), you are right: Being able to continue uploading 10'000 tapes after it failed at tape ~9'875 would really be a good thing! (And being able to upload those tapes you solved when sitting offline in an airplane, would also be a cool thing!)

Thanks again for your ideas and recommendations! :-)
Post Reply