Rocks'n'Diamonds

i'm not sure how practical this would be to implement, but i'll describe it anyway.

so the more levelsets you have, the longer it takes to load, of course.
i have both the Boulder Dash levelset pack and the Emerald Mine Club levelset pack installed, which are pretty large and include custom artwork and sounds. loading times vary, but recently it took about 1 minute for rocks'n'diamonds to finish loading, even though i just wanted to test some BD stuff in the level editor.

then i thought: what if you could choose to skip loading folders of levelsets you don't want to play? maybe, whenever the loading process is in the middle of loading levelsets from a folder (e.g. Emerald_Mine_Club, Boulder_Dash), you could click or press a key to skip loading the rest of the levelsets from that folder, and move onto the next folder (or next standalone levelset). then, if you go into the levelset selector and try to access a folder that you skipped loading, it would pick up from where it previously stopped loading.

the one extra complication i can think of -- though i'm sure there are many more in practice -- is if a folder you want to skip contains the levelset the game is set to start on. in that case maybe disallow skipping that folder, or have skipping it just load the levelset the game starts on without loading the rest (and when that folder is loaded again, skip over loading the levelset that's already loaded). the former sounds easier.

Yes... this is a good example of something that was no issue for a long time (only a few dozen level sets in R'n'D), but at some time cannot be ignored anymore (several large level collections, namely the large EMC and BD collections with hundreds of level sets each, causing really noticeable loading times now, especially on a cold disk cache).

Btw, this was even worse years ago, when not only the whole levels directory was recursively read on program start, but also the "levelinfo.conf" file for each level set was parsed (which is now cached in a cache file).

The main reason why everything is loaded every time is because level sets may reference artwork sets (which itself may be part of other level sets) for which the location in the levels directory must be known when loading such level sets.

However, there is most probably a lot of potential for optimization. (And I think it should not require user interaction as you have described it, which at best would be a workaround.)

I will have to think about how this could all be done better!

I think you could probably have a single(*) on-disk cache file for the entire discovery process, instead of looking for and opening hundreds-to-thousands of directories and files. Only run the full discovery process when needed.
(*) Or a few, however many is reasonable...

When needed? Well, the game now has internal ways to add levelsets (I think -- I haven't really played with that stuff much, but can we download levelsets from the score server now? Drag and drop zip files into the game?!) -- any time that sort of stuff happens, either patch the internal data structures and write them back out to the cache; or run the full discovery, if the internal state is too fragile to update on the fly.

Any time you're accessing stuff that the cache says should be there, if a file is missing or has wrong contents, prompt the user that a refresh is needed (then run the whole discovery).

Store file timestamps in the cache, and check them on open, as part of this hinting process.

Provide an in-menu way to manually trigger the process -- for people who poke around at files by hand.