where did Element Properties go?

Discussion around programming R'n'D, its source code and its tools.

Moderators: Zomis, Flumminator

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

Re: where did Element Properties go?

Post by filbo » Thu Mar 26, 2015 7:12 am

I've done `git pull` many times (just now it got some changes including an intriguing "added step-based engine snapshots to undo/redo single steps in game"?? -- before that it was quiet for a few days).

I run ./rocksndiamonds out of the git repository / build directory, while having an old RnD archive splayed out on the system. I build with RO_GAME_DIR=/usr/share/games/rocksndiamonds RW_GAME_DIR=/var/games/rocksndiamonds. So perhaps some graphics should be pulled out of '.' but are instead pulled from the old splayed out archive?

I do it this way because I have all the levels downloaded and in place; all the score files etc. shared across my account and my kids'. I don't want to have a whole separate copy of all those files for use with dev binaries...

> Regarding the unsolvable level: I think I will fix it at least in the DX BD set...

I only have notes on the emc_bd4/ levels, of which 18, 42, 47 and 80 are not solvable (18 needs about 20s more on the clock; 42's exit is unreachable due to wild amoeba growth; 47 & 80 need horizontal wrap).

I have notes on a total of 95 unplayable levels, all in the EMC area (where I've been playing lately). I only started taking notes about 8 months ago. Some of the older notes are just "unplayable", so before I expose the whole set I'd want to go back and note why...

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

Re: where did Element Properties go?

Post by filbo » Thu Mar 26, 2015 7:31 am

Ok, I needed to copy graphics/gfx_classic/*.png to /usr/share/games/rocksndiamonds/graphics/gfx_classic/.

The old ones were from February and had the 2013 copyright date.

Running the latest binary on an EMC level (emc_beer_mine):

I play a lot in single-step mode. Every step, I see the large 3-button tape controls flicker between "stop-pause-play" (yellow on black) and "step backwards-pause-step forwards" (white on grey). This is a bit ugly and distracting. I see it's work in progress.

After some play it failed with:

| rocksndiamonds: fatal error: cannot allocate 251016 bytes -- out of memory
| rocksndiamonds: aborting
| libgcc_s.so.1 must be installed for pthread_cancel to work

Am I stressing its memory allocation strategy if I play hundreds of steps of single-step? Hmm, no, I see I can back-step even if I was playing in realtime.

Hmmm, just playing the same tape back repeatedly, I can see the process size growing without limit. Until it hits my `ulimit -v 5000000` (roughly 5 gigabytes) and dies (much more gracefully than does Firefox, which is the process I'm really trying to limit...)

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

single-stepping memory armageddon :)

Post by filbo » Thu Mar 26, 2015 7:40 am

Process grows about 1.6MB per player move. Probably level size-dependent: this is an EMC level of 64 x 32 dimensions.

That's 2K cells. What's consuming 800 bytes per cell per move?!

In any case, it needs to be deleted when that run goes away (when user starts playing same or different level from scratch).

Or when part of a run gets pinched off: if I play for a while, then single-step backwards a bunch, then resume playing (taking different actions from last time), my previous set of actions is now dead. I can see from memory usage that it isn't deallocated.

Ok, will stop complaining about fresh hot in-development stuff that you probably have fully under control...

User avatar
Holger
Site Admin
Posts: 3223
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: where did Element Properties go?

Post by Holger » Fri Mar 27, 2015 8:16 am

Sorry for the memory problem -- it should free it all with each new game, but I can confirm that it doesn't. WIll have a look at it and try to fix that as soon as possible.

And yes, it's not optimized at all regarding memory consumption -- for example, each step snapshot also contains lots of internal data for custom element handling for each playfield cell/tile, even if a level does not use them at all. This was no problem for F1/F2 style (single) snapshots, but needs further optimization for step-based snapshots, of course.

User avatar
Holger
Site Admin
Posts: 3223
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: where did Element Properties go?

Post by Holger » Sat Mar 28, 2015 3:25 pm

Memory hole (broken deallocation of allocated snapshot memory) fixed now. Very stupid error which existed for a long time, but apparently did not show up because a lot less snapshots were taken before during program execution.

Those new step snapshots still have to be space-optimized before they can go live, though...

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

Re: where did Element Properties go?

Post by filbo » Sun Mar 29, 2015 8:55 pm

Thanks! That makes the game playable again, at least for levels of only a few minutes' play duration.

Boy, the original Boulder Dash ran happily on machines which were thrilled if they ever got upgraded to 64KiB RAM :)

User avatar
Holger
Site Admin
Posts: 3223
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: where did Element Properties go?

Post by Holger » Wed Apr 01, 2015 5:45 pm

> I play a lot in single-step mode. Every step, I see the large 3-button tape controls flicker between "stop-pause-play" (yellow
> on black) and "step backwards-pause-step forwards" (white on grey). This is a bit ugly and distracting. I see it's work in
> progress.

Fixed.

> That makes the game playable again, at least for levels of only a few minutes' play duration.

The default step-based engine snapshot mode mode is now "every move" instead of "every step" (which it was before and can still be configured in setup menu now, plus completely disabling it). This will decrease RAM usage during the game a bit (although it does not make a difference in single-step mode). Nevertheless, I think I will add a check for available free memory, and set the default mode for step-based engine snapshots to "off" if it is less than a certain amount. Besides that, it should be possible to decrease per-snapshot RAM usage a little bit (by not saving custom element states for non-CE levels, for example), and to add some less memory-hungry modes like "last 100 steps/moves" or "last 100 seconds" or something like that...

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

Re: where did Element Properties go?

Post by filbo » Thu Apr 02, 2015 12:05 am

"Last N stuff" would be a good memory-saving mode.

Even better: automatically thin the ranks to maintain a sensible memory profile.

Keep an eye on memory size; don't know if threshold should be user-settable or guessed in some way from system-at-hand (TBD). If you have calls like getrlimit(RLIMIT_DATA) -- presumably you do on each OS, though they're different -- use that to set a maximum for the user tunable.

When getting too big:

If saving every step: delete every other step, starting from beginning of recording and ending some seconds before current head.

If saving every move: delete every other move, etc.

If saving every second: delete every other second, etc.

I think you don't even have to remember whether you've done this before; the effect of repeating it is good. What I visualize:

Let's imagine there's room for 5000 snapshots in memory. Doesn't matter what you're saving, but let's say "step" for purposes of exposition.

So I play for a while, until 5000 steps are snapshotted; now we're out of memory. Suppose steps are accumulating at a rate of 50/sec, so this is 100s worth of play. Also suppose that N, "some seconds" mentioned above, is 5. So we currently have in memory steps from t=0 to t=50 and step=0 to step=5000 (ignoring the off-by-1s).

Then thin out every other step from t=[0..45]; which means steps [0, 2, 4 ... 4500] are deleted. In time terms, t[0..45] is half-remembered while t[45..50] is fully remembered. And we got rid of 2250 snapshots.

So now we run out again at t=72.5. Now we thin out every other, meaning steps [1, 5, 9, ... 4497] by original numbering, and steps [4502, 4504 .. 6750], are forgotten. Now, in time terms, t[0..45] is 1/4-remembered, t[45..67.5] is half-remembered, and t[67.5..72.5] is fully remembered.

etc.

That sounds complex, but in actuality you would have compressed the array so what's really going on in each round is thinning out every other item; you don't have to remember the chop points for the different thinning thresholds because there are none. Each remaining record includes a time stamp or step count or whatever, so you can still play back successfully by stepping through what's present.

This is partly based on thinking that high detail is more useful at the current tip of action.

Also, in actuality, you probably wouldn't "free" the thinned out records (no calls to system library allocator/deallocator), nor shuffle down the rest; instead you would keep an indirection table, and mark thinned out entries as free. So a "thinning event" shouldn't take a lot of effort and shouldn't cause a stutter in play.

Sorry for somewhat stream-of-consciousness design ideas :) Let me know if it's unclear...

User avatar
Holger
Site Admin
Posts: 3223
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: where did Element Properties go?

Post by Holger » Tue Apr 07, 2015 2:55 pm

This indeed sounds like a very good strategy to save memory while doing undo snapshots without artificially limiting the number of snapshots to the last 50 or 100 steps or moves.

The main problem seems to be to detect the amount of free, non-swap memory (in a platform-independent way) available for the R'n'D process, but even without this (as you mentioned), setting the limit to, say, 100 snapshots and use that strategy would be a good thing to have snapshots available from the start of the game to the last step just made.

filbo
Posts: 357
Joined: Fri Jun 20, 2014 10:06 am

Re: where did Element Properties go?

Post by filbo » Wed Apr 08, 2015 6:24 am

I don't think you can do it in a platform-independent way, but you're already aware of what platform you're being built for, so that makes little difference.

100 sounds pretty sharply constrained; maybe 1000? As a default on platforms where you don't yet have a better way of knowing.

The better (platform-specific) way might give you an answer larger or smaller than the default. Believe it if it's low. Maybe have a user setting for max # frames to save -- defaulted to 1000. Then the actual number you save is min(user setting, what OS says [1000 if no OS-specific call, or it fails]). And actually the default might also be platform specific e.g. perhaps smaller on Android (not sure what the constraints really are...)

User avatar
Holger
Site Admin
Posts: 3223
Joined: Fri Jun 18, 2004 4:13 pm
Location: Germany
Contact:

Re: where did Element Properties go?

Post by Holger » Tue Jun 14, 2016 3:06 pm

The current code in the Git repo now contains the automatic engine snapshot de-allocation strategy more or less just as you described it. I've decided to limit the maximum number of snapshots not by frames (snapshots), as this might differ heavily depending on level type (with or without custom elements, even though it currently stores "everything", so this still has to be optimized), but by overall size of all stored snapshots in bytes. So if the maximum memory size is reached, about half of the snapshots will be removed (keeping the "newest" 10 % and removing every second snapshot of the remaining "oldest" 90 % snapshots in the list). This seems to work just fine.

The maximum memory size for engine snapshots is set to 512 MB by default (which currently is enough to store a few hundred non-space-optimized snapshots and should be fine for most not-too-old systems). It can be changed in the file "setup.conf" using the option "engine_snapshot_memory" (default: 536870912 bytes).

Post Reply