Excessive Memory Consumption

Install & setup problems, application questions and other nasty things ;)

Moderators: Frustikus, simX, Aus-RED-5

Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Excessive Memory Consumption

Postby Sirius » Thu May 29, 2014 4:45 pm

Hmm. I am running into those memory issues too. In 32-bit even the simplest levels don't load; 64-bit they do but seem to start at 3.5 GB working set, and by the time you get to something like the single-level version of Dimensional Intersect, even that's not enough.

Looks like it's time to refresh my checkout of the XL SVN and go looking for memory debug tools :) I know you've been working on memory earlier so it looks like what I've got may not happen for everyone.
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Excessive Memory Consumption

Postby karx11erx » Thu May 29, 2014 4:47 pm

Are you using the latest D2X-XL software?
Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Re: Development Blah: Comments

Postby Sirius » Fri May 30, 2014 3:41 am

1.17.21 at the time I posted. Already out of date, though.
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Development Blah: Comments

Postby karx11erx » Fri May 30, 2014 9:11 am

Well, I had fixed these memory issues in an older version already, and I cannot observe any here (using the 64 bit version, but memory consumption is memory consumption).
Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Re: Development Blah: Comments

Postby Sirius » Fri May 30, 2014 4:27 pm

OK, nearly to building, looks like there are some missing libraries by now. I have zlib but it's an old version (1.2.3) - I don't know if anything very important changed but the D2X-XL .sln now points to 1.2.5. Then there's Rift, which I don't have at all. Everything else I still have copies of.
Do you keep copies of these somewhere on the site that I've missed? Otherwise I'm sure I can find them, I just need to generate .vcxproj files for them I suspect...
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Development Blah: Comments

Postby karx11erx » Fri May 30, 2014 5:31 pm

I will check out the libs I am using and update the archive on my server if necessary. I will include my Rift libs.
User avatar
Weyrman
D2X-XL Expert
D2X-XL Expert
Posts: 579
Joined: Tue Sep 25, 2007 7:47 pm
Location: Brisbane Australia
Contact:

Re: Development Blah: Comments

Postby Weyrman » Sat May 31, 2014 12:24 am

I am pleased to be able to say that I have levels loading in 1.17.23 x64 running with hi-res textures and models switched on.

My 3.4Ghz i7 runs with all 8 cpu graphs running at 100% and 6.5gb of memory in use but I played! With anaglyph 3d glasses as well.

Its good to be back. :D Thanks to your compulsive bug hunting :away: :hammer:
-SYSTEM SPECS-
MBoard:"GigaByte Z68X-UD3H-B3" Motherboard
CPU: Intel i7 E2600 3.40GHz
RAM: 8GB DDR3 2133MHZ
HD: WDC 10EARX 1TB SATA
Video: GigaByte 1G ATI 4890
Monitor: Asus VS428 LCD 24"
Sound: Onboard - now with connected 5.1 sound system
OS: Windows 7 x64 Home Premium
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Development Blah: Comments

Postby karx11erx » Sat May 31, 2014 12:58 am

I don't quite understand that huge memory consumption you and Sirius seem to be having. I am not encountering any such problems at all. :think:
Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Re: Development Blah: Comments

Postby Sirius » Sat May 31, 2014 8:01 am

OK. It builds, it runs, and I still get the error; now comes the tricky bit, figuring out why...
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Development Blah: Comments

Postby karx11erx » Sat May 31, 2014 9:00 am

I have tried several levels with the upcoming D2X-XL v1.17.24 today, and it never used more than 1 GB total, and only around 400 KB max. for a level. The other 600 KB are basic memory consumption just for the menu system and for hires models (which use most of that memory).

Weyrman wrote:your compulsive bug hunting

You got me. :roll:
User avatar
Weyrman
D2X-XL Expert
D2X-XL Expert
Posts: 579
Joined: Tue Sep 25, 2007 7:47 pm
Location: Brisbane Australia
Contact:

Re: Development Blah: Comments

Postby Weyrman » Sun Jun 01, 2014 2:52 am

1.17.24 x64 with all hi-res - CPU 100% 6.86GB Memory

after shutdown the computer runs at 2.5GB

:jump: BUT IT RUNS :jump:

Does d2x-xl x64 need a chunk of continuous memory to run? I tried to load a level to check something and it just wouldn't load, kept dying with an out of memory error. Shut the program down and did something else and then had another go and the level loaded fine. I often do reboot before trying to run x64 xl because there is a better chance of it running first go.

Question: what settings do you have in the render screen?

Framecap - 120 FPS
Lighting - High
Lightmap Quality - Basic
Lightmap Precision - Standard
Colored Light - Full
Image Quality - Med
Render Quality - Med
Cameras - Basic
3D Powerups - Basic
-SYSTEM SPECS-
MBoard:"GigaByte Z68X-UD3H-B3" Motherboard
CPU: Intel i7 E2600 3.40GHz
RAM: 8GB DDR3 2133MHZ
HD: WDC 10EARX 1TB SATA
Video: GigaByte 1G ATI 4890
Monitor: Asus VS428 LCD 24"
Sound: Onboard - now with connected 5.1 sound system
OS: Windows 7 x64 Home Premium
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Development Blah: Comments

Postby karx11erx » Sun Jun 01, 2014 3:11 pm

I used your player profile from your last bug ticket, and I couldn't see any excessive memory consumption like you do.

My own gfx settings are all max'd out (except lightmaps - I am using the "high" setting for them), and I don't have any such problems with any level I load either.

Edit:

No problems with 32 bit version (multi threaded) as well.
Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Re: Excessive Memory Consumption

Postby Sirius » Mon Jun 02, 2014 3:36 am

There were a few things I noticed about this:
1) Scaling down settings seemingly didn't help at all. It happens on low display settings with lightmaps off, effects off, and so on.
2) It's probably not a memory leak - almost all the additional memory used goes away after the load fails, or if it doesn't, after you exit the level.
3) The extra memory is retained while the level is running - it's only freed upon returning to the main menu.

Unfortunately finding a memory profiler that actually works with D2X-XL is not particularly easy. :( If it were a memory leak there are plenty of tools I could use, but this limits options a bit.
Sirius
Posts: 1990
Joined: Tue Sep 25, 2007 10:29 pm
Location: Bellevue, WA
Contact:

Re: Excessive Memory Consumption

Postby Sirius » Mon Jun 02, 2014 4:39 am

OK, thanks to certain breakpoint features I managed to get enough information to have some idea - I attached it to a new bug here:
https://sourceforge.net/p/d2x-xl/bugs/2735/

Figuring out exactly what's going on might not be too quick for me which is why I thought it would be better to share my results so far. I'll look into why the allocations are so large as time permits though. I should get back to the DLE work before I need to sleep :)
User avatar
karx11erx
D2X-XL Master
D2X-XL Master
Posts: 8112
Joined: Mon Sep 24, 2007 8:48 pm
Location: Wilferdingen, Germany
Contact:

Re: Excessive Memory Consumption

Postby karx11erx » Mon Jun 02, 2014 11:11 am

d2x-xl bug tracker wrote:You need to look into ReadHiresBitmap(). A good start would be to make it print the names of the bitmaps it loads to the log file (using the PrintLog () function).

I have the suspicion that your source code isn't up to date. As I said, there had been a bug where D2X-XL loaded the entire hires animation for each frame of that animation. That is because for the original animated bitmaps, each frame is handled and stored as a separate texture, and Descent only figures they are part of an animation by some extra information. Part of this information is the animation frame number in the texture name, preceeded by a hash sign ("#"). Additionally, Descent has information about each animated texture. That information basically consists of the texture indices (which simply point into Descent's texture list), animation frame times, and a few more things.

To make hires animation handling more convenient for texture artists, I decided to have all frames of such an animation in a single texture. D2X-XL handles hires texture loading as follows: For the level to be loaded, it loads all low res textures. For each such texture, it tries to find the corresponding hires texture. Now animations are a special case: D2X-XL should only load the corresponding hires texture for the animations first frame (let's say "door01#0"). It must not load hires textures for the other low res door animation textures ("door01#1", "door01#2", etc.). Instead, D2X-XL will create hires child textures referring to the hires animation, each of which just describes one frame of that texture, and will assign these child textures to the low res animation frame textures.

Now what had had happened (until I had fixed that bug) was that D2X-XL loaded the entire hires animation for each low res animation frame (e.g. not just for "door01#0", but also for "door01#1", "door01#2", etc.), which ofc resulted in a massive memory consumption.

Now both you and Albie still have that problem, while I do not. This makes me wonder whether both of you built the D2X-XL executable yourselves from the source code and probably are using some outdated source files.

So my next question would be: do you have that problem with the executables I have built and stored on my web server and SF.net, too?

Return to “D2X-XL Help”

Who is online

Users browsing this forum: No registered users and 5 guests