Page 1 of 43 1 2 3 4 5 6 7 8 9 10 11 26 ... LastLast
Results 1 to 20 of 858

Thread: "Insufficient memory" & "CTD" - workaround

  1. #1
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default "Insufficient memory" & "CTD" - workaround

    It's only a matter of time before this becomes common knowledge, I hinted at this issue for a couple days already, so I wanted to handle this subject responsibly because it is something I am well versed in.

    Do you get "Insufficient Video Memory" crashes?
    How about repeated crashes to desktop precluded by everything slowing down and feeling sluggish?


    These are obvious signs of running out of virtual address space.

    You may even get a crash after several battles or a another repeated crash due to this specific cause.

    I'll keep this article basic and in laymens terms with a little explanation, you can spend a week researching it to begin getting a deep understanding of it, so I will keep things simple since I have done the heavy lifting for you.

    BACKGROUND (explains the issue)

    What is Virtual Addressing?
    Virtual Address Space is a set of numerical addresses that windows maps and keeps track of for memory usage. It has nothing to do with how much RAM you have or how much virtual memory you have. There is only a fixed amount of addresses available for a 32 bit application like empire and that is 2GB, regardless if you are on a 32 bit or 64 bit OS. A 332 bit OS itself has 2GB available in "user" address space, with 2GB normally called "kernel" address space. In addition to this, and irrespective of this, a 32 bit application is limited to 2GB for it's address space determined by the binary header (this is true for 32 bit and 64 bit OS's alike). A person cannot track virtual address space with tools like task manager.

    Why are you affected when others are not?
    Each machine maps different amounts of virtual addresses on start up. It's affected by many things. Large impact to virtual addresses being reserved, and therefore becoming unavailable, is due to things like Vista, large memory video cards, multiple devices whose drivers reserve addresses on boot up, etc. These reserved addresses can be viewed in ones device manager.


    Why is Empire affected by this issue?
    Quite simply it uses a lot of memory and we are still in predominantly a 32 bit world. Addresses are limited and Vista and high memory video cards reserve and make addresses unavailable. CA chose not to utilize 64 bit safe coding practices OR natively support 64 bit so people with 64 bit OS's are also affected by this issue.
    This is predominantly a 32 bit problem, as game assets grow then memory footprints grow. Also people with large video ram (>768) usually have no idea that they are cutting their available address space that video applications can use. It's why 1GB video RAM is not as common as one would think or is never fully utilized used by devs, because the operating system cannot handle really it due to limited addresses.

    This three part series on a "Messy Transition" at Anandtech was born from a case study of a memory address workaround I made for Supreme Commander, similar to the one I am making here, and it is a great series to explain the issues and brought alot to the surface for all to see.

    Part 1 -http://www.anandtech.com/gadgets/showdoc.aspx?i=3034
    Part 2 -http://www.anandtech.com/cpuchipsets...oc.aspx?i=3044
    Part 3 -http://www.anandtech.com/systems/showdoc.aspx?i=3060

    What can the developer do?
    Two things primarily. Lower memory usage by the game, that will likely be the first goal but that may not be enough for people who's machines reserve alot of addresses on startup for the video card. Secondly, they can make the game 64 bit safe allowing people with 64 bit to immediately bypass this issue. While 32 bit users can then extend the OS's user address space, not optimal, but likely the best solution we will have unless they drastically reduce memory usage and management in game. Making the code 64 bit safe is unlikely because of the work involved and would have been much easier to do properly from the beginning. CA have a full plate and have many bugs to address that need attention so developer resources and money is limited, that is why the latter option is doubtful.
    For the foreseeable future this workaround should help if no real progress is made otherwise, remember hitting the 32 bit limit is not hard and many games do.
    EDIT: According to a post on this page below, the developers are indeed preparing a 64 bit "safe", not necessarily native 64 bit, Large Address Aware executable that is currently in testing. So this is encouraging and is a pleasant surprise from what I expected, definitely good news.
    EDIT: This update to the executable has arrived on the 4-29-09 patch.

    THE WORKAROUND
    (the details)

    I am providing a workaround similar to the one I provided for Supreme Commander back in the day here for reference.

    What does it do?
    It has to do two things that both rely on each other to work. For 32 bit operating systems we have to tell the operating system it can use more than 2GB user address space, 2.5GB actually, while leaving 1.5Gb for the kernel. We also have to set the Empire.exe executable to be allowed to address beyond the 2GB ceiling. 64 bit OS users don't have to do anything.

    What does it help with?
    This is not the solution for all crashes. It deals primarily with the crash to desktop that is caused by lack of virtual address space.Due to the unique nature of how addresses are mapped differently on each machine this crash may happen at different places and times of the game, even loading a map, menu or waiting on an AI turn. If you try this fix, and you follow directions properly, and it doesn't help you then your crash is due to something else.

    How do I do it?
    Follow directions explicitly. I have seen people consistently skip steps or not read the directions properly so it doesn't work for them whne they don't do what it says. it's not as complicated as it sounds, I'm just being thorough.

    APPLY THE WORKAROUND
    (juicy caramel center)

    Requirements

    Only real requirements is that you have atleast 2GB RAM. You don't need beyond that even though we are allowing addressing up to 2.5 GB. Also following instructions and being somewhat familiar with working with files in windows is required. Make sure you are using Vista SP1. If not then you need this hotfix that fixes a major blunder with Vista address space - http://support.microsoft.com/kb/940105
    That's all, follow the 3 steps.

    1) Prepare the operating system to be able to use beyone 2GB address space with a userva of 2.5GB (only 32 bit OS's need to do this): I am purposely using only 2500 for userva for specific reasons and any more will likely not be needed, as a result the kernel will have plenty of space to play in.

    All 64 bit OS's
    Nothing to do on the OS.
    Proceed to step 2 on setting the executable. 64 bit OS's running 32 bit binaries are also limited to 2GB application space.

    XP 32
    * Find your Boot.ini located usually on the C:.
    If you cannot see it, you probably have system files hidden, then follow these instructions at microsoft (also uncheck "hide protected operating system files")
    * Right click on the boot.ini and choose properties. Uncheck "read only"
    * Open the boot.ini by double clicking it.
    * Under the section called [operating systems] find the line that says "multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Professional...". Yours may look a bit different. We are going to make a duplicate, so that if you make a mistake you will not harm yourself from booting into your computer. Drag you're mouse across the entire line, right click and choose copy. Then right below the line you just copied right click your mouse and choose paste. You should now have two lines one after another that are exact duplicates.
    * Change the bottom line that you just copied with the following addition on the end. Add a space at the end of the line and then this without the quotes "/3GB /USERVA=2500"
    * In that lower boot entry you just made, find the name of the operating system in quotes and change it to something like "Windows XP 3GB", so you can differentiate at boot up.
    * Close and save the boot.ini.
    * Right click on the boot.ini, hit properties, check read only.

    Vista 32, Win 7 32bit
    * In Vista go to Start menu. Go to Accessories. Locate the command prompt shortcut and hover mouse over it. Right Click on the shortcut then select Run as Administrator.
    * In the command prompt type this exactly: bcdedit /set increaseuserva 2500. Then hit enter.
    *
    Make sure you get a message back confirming the change was made. To verify the entry is there you can type just bcdedit, hit enter, and you should see the entry now listed.
    * Then close the command prompt. You just told Vista to increase user virtual address (userva) space to 2500MB.
    * Changes take effect on reboot but don't reboot yet.


    SUMMARY
    We extended the user virtual address space in XP and Vista to 2500MB, so now applications can utilize up to that much if they are set to be Large Address Aware. While some 32 bit games/applications are set, Empire is not.

    2) Now reboot the PC because Windows needs to set the userva at 2500 which only happens after startup.

    If you skip any step it will not work. When you have rebooted you should be good to go. Run the game as normal with the original game shortcut.

    TO UNDO THE CHANGES

    For the XP 32 bit just delete the line from your boot.ini.
    For Vista 32 Open command prompt as administrator and type: bcdedit /deletevalue increaseuserva. That deletes the entry.
    Reboot and you are back to normal.

    THIS IS NOT A FIX
    No it is not. We should still continue to put pressure on the developers to fix this issue. A bad side effect of user made workarounds, while they actually help people be able to play the game which is important, the problem is that devs stop hearing about the issue by a large group so they think the impact isn't big. So continue to update the issues and CTD's you have to the official boards. One such topic exists here on the official boards: http://shoguntotalwar.yuku.com/topic/49357

    POSSIBLE ISSUES
    There are two side effects to watch out for and one addendum.
    1)
    A very small percentage of machines may have issue booting due to a badly written driver after making the change to the OS in addressing above 2GB because drivers utilize higher addresses. To fix this boot problem, with XP you have the other boot option that appears at boot to choose. With Vista go into safe mode and follow the undo instructions above on deleting the userva entry. This workaround will not work if you have a bad driver affecting your boot up.

    I'll edit the topic as necessary. Post your experiences, I will try and help if can with specific issues on this workaround.

    OTHER REFERENCES (incase the wall of text wasn't enough)
    Windows memory limits in 32 and 64 bit:
    http://msdn.microsoft.com/en-us/library/aa366778.aspx
    A bit about memory addressing and 3GB switch:
    http://www.microsoft.com/whdc/system...AE/PAEmem.mspx


    Thanks to TheSpaceHamster/TheCookieMonster for testing this and starting a thread in the official tech support that got my attention.
    Last edited by irishron; January 10, 2012 at 07:05 PM. Reason: 4-29 patch

  2. #2
    Tassen's Avatar Yoda
    Join Date
    Feb 2009
    Location
    Chicago - Illinois - USA
    Posts
    553
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    I am lucky enough yet to experience any of all these CTDs others do and I read about every day. However, since this 'fix' is such a quick and easy one to apply I will do it as a preemptive means of preventing any CTDs caused by insufficient memory.


    Thank you posting this and it should help many others who's game experience has been, less than it should be, thanks to this error.

  3. #3
    Ikko-Ikki
    Join Date
    Mar 2009
    Location
    Geneva, Switzerland
    Posts
    23
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    This is the only problem I have with the game. Thanks for sharing, much appreciated!

    I will post tomorrow if it worked for me.

  4. #4
    jmbuehler's Avatar Signifer
    Join Date
    Dec 2008
    Posts
    196
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    MadBoris you rock! Again, that is... Will give it a shot tomorrow.

  5. #5
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by jmbuehler View Post
    MadBoris you rock! Again, that is... Will give it a shot tomorrow.
    From the other topic, I heard you had an issue with the boot, but maybe lessening the userva to 2500 or less will help this time around, but may not. Give it a go, but 3 GB vid cards is tough. As for your SLI question from the other topic, I have no idea. Bleeding edge sometimes cuts deep.
    Last edited by MadBoris; March 16, 2009 at 10:01 PM.

  6. #6
    Spaxspore's Avatar Pili Prior
    Join Date
    Dec 2007
    Location
    USA
    Posts
    2,011
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    thanks so much, ill give it a shot.

  7. #7
    Tassen's Avatar Yoda
    Join Date
    Feb 2009
    Location
    Chicago - Illinois - USA
    Posts
    553
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Ok, after applying the patch, the changes to boot.ini and at reboot Windows did let me back in again...at 800x600 4-bit. It was NOT a pretty sight.


    What gives? Crappy video driver? I am currently on the 185.20 beta drivers.

  8. #8
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by Tassen View Post
    Ok, after applying the patch, the changes to boot.ini and at reboot Windows did let me back in again...at 800x600 4-bit. It was NOT a pretty sight.
    Maybe more likely a motherboard driver. Myself, I have been using /3Gb switch for a couple years with no issue. Never any issues with vid drivers in XP and I go through nvidia drivers often. Although I never used that driver but it's doubtful. it's likely something else entirely.

    It's hard to identify the culprit and I am hesitant to start suggesting workarounds for a workaround. Although latest motherboard drivers may address the issue. Boot in normally, update the driver to the latest if you like then see if there is any benefit.

    What mobo chipset you got?

    Edit: There will be certain hampered by this but in the past experience has been few, although maybe certain chipsets and their drivers have an issue here.
    Last edited by MadBoris; March 16, 2009 at 10:16 PM.

  9. #9
    Tassen's Avatar Yoda
    Join Date
    Feb 2009
    Location
    Chicago - Illinois - USA
    Posts
    553
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    My mobo is the ASUS A8N-32 and the chipset is the NVIDIA nForce4 SLI X16.

  10. #10
    TheSpaceHamster's Avatar Wimmer
    Join Date
    Feb 2009
    Posts
    133
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Great works and help MadBoris. Things have still been running great for the past few days, but you are 100% correct; this is a work around, we must still pressure the development team to properly optimize their game and code for thorough gaming use.
    Go for the eyes Boo! The eyes!

    "Boo! Oh, you've been a naughty hamster, mister! Don't you ever scare Minsc like that again!"

    AKA TheCookieMonster @ http://shoguntotalwar.yuku.com/directory

  11. #11
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by TheSpaceHamster View Post
    Great works and help MadBoris. Things have still been running great for the past few days, but you are 100% correct; this is a work around, we must still pressure the development team to properly optimize their game and code for thorough gaming use.
    Well there is our first success story. SpaceHamster is the guy who started a insufficient memory crash thread at tech support , now "that crash" isn't affecting him after a couple days of testing this workaround. Good that there are no other crashes you have run into, it's a bummer when people solve one crash only to be hit by a different one, which happens unfortunately.

    Thx for the feedback and for pushing the issue.
    Last edited by MadBoris; March 16, 2009 at 10:43 PM.

  12. #12
    arnel98's Avatar Murakawa
    Join Date
    Nov 2006
    Posts
    21
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    i guess this issue wont completely go away until games are written on 64 bit only with no 32-bit compatiiblity...
    I mean, with the amount of stuff they put on games now, there will come a point where no type of optimization will allow them to make gmaes that would fit on the 2gb virtual address limit.

    I think I first experienced this issue on Supreme Commander. After that game, found the bcdedit techinique on google, set my setting to use 2700 (for the memory limit) and never looked back. I tried 3072 before but some of my drivers become unstable so in my case, 2700 seems to do the trick.

  13. #13
    Spaxspore's Avatar Pili Prior
    Join Date
    Dec 2007
    Location
    USA
    Posts
    2,011
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    i just played my Prussian campaign turn 189+... played 5 battles and no crashs after this patch.

    Thanks so much Madboris +rep

  14. #14
    Chinen
    Join Date
    Jan 2009
    Location
    Sydney, Australia.
    Posts
    446
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Good guide MadBoris. However this will do nothing to stop the more frequent VRAM overflow crash, unfortunately.
    E8500 @ 4GHz (445x9) | TRUE120 + Noctua NF-12P | 4GB 1066MHz Corsair Dominator (5-5-5-15) | Gigabyte EP45-DS4P | HIS HD4870x2 | TT Truepower 750w Modular | TT Armor+ | 2 x WD 640GB | Windows 7 RC 64-bit

  15. #15
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by R4nd0M View Post
    Good guide MadBoris. However this will do nothing to stop the more frequent VRAM overflow crash, unfortunately.
    Which one is that? I take it you don't mean "Insufficient Video Memory Error" which is what this is geared to address?
    I guess I hadn't heard about the "VRAM overflow error".

    Yes, this is by no means a cure all. But I do think people may not realize the extent that several diverse crashes, with different messages, may in fact lead right back to this cause. It all depends what the game is doing when the memory addresses dry up to what message or what kind of a CTD is experienced. Time will tell as more people test it out for their particular repeated consistent crash.
    CTD's in games can come from various causes though, of course, some of which are strictly coding errors.
    Last edited by MadBoris; March 17, 2009 at 02:17 AM.

  16. #16
    Chinen
    Join Date
    Jan 2009
    Location
    Sydney, Australia.
    Posts
    446
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by MadBoris View Post
    Which one is that? I take it you don't mean "Insufficient Video Memory Error" which is what this is geared to address?

    Yes, this is by no means a cure all. But I do think people may not realize the extent that several diverse crashes, with different messages, may in fact lead right back to this cause. It all depends what the game is doing when the memory addresses dry up to what message or what kind of a CTD is experienced. Time will tell as more people test it out for their particular repeated consistent crash.
    CTD's in games can come from various causes though, of course.
    This guide is definitely a step in the right direction from a user standpoint. I don't need to tell you that adjusting the memory space ETW can address the the leaks in normal system memory and improve performance overall.

    However I'll talk from the experience of having run ETW in a 64-bit environment, with the application header tweaked to use a large address space and still experiencing (two, today in fact) "Insufficient Video Memory Error" crashes.

    I'll pin that on the second part of your above reply. We don't know how CA is allocating resources, and I won't argue with those who have not crashed while using this fix.
    E8500 @ 4GHz (445x9) | TRUE120 + Noctua NF-12P | 4GB 1066MHz Corsair Dominator (5-5-5-15) | Gigabyte EP45-DS4P | HIS HD4870x2 | TT Truepower 750w Modular | TT Armor+ | 2 x WD 640GB | Windows 7 RC 64-bit

  17. #17
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by R4nd0M View Post
    with the application header tweaked to use a large address space and still experiencing (two, today in fact) "Insufficient Video Memory Error" crashes.
    If you verified the header on the exe in the game folder, like my batch makes easy, then that may just mean there are more causes than just one for that error message.
    Thx for the feedback.
    Last edited by MadBoris; March 17, 2009 at 02:56 AM.

  18. #18
    Creative Assembly
    Join Date
    Sep 2005
    Posts
    15
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    A very good summary of the issue MadBoris. At some point in the near future we'll start releasing ETW executables linked with /LARGEADDRESSAWARE enabled, it's currently in test. This will at least save you the second part of the process (and having to repeat it!).

    Just to expand a little further, the reason why this can cause both 'out of video memory' and 'out of system memory' issues is because DirectX grabs a fair amount of application address space for its buffers, and these resources are placed inside the user portion of the full 4 mb range. The memory issues are certainly not responsible for all problems people are reporting - for example crashes during the end-of-turn are down to other things, and there are a few other cases which can throw up identical errors - but it's one of several priority items for the programming team at CA right now.

    +1 rep
    Last edited by JeromeGrasdyke; March 17, 2009 at 04:13 AM.

  19. #19
    Yoshihara
    Join Date
    Feb 2009
    Posts
    78
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by JeromeGrasdyke View Post
    A very good summary of the issue MadBoris. At some point in the near future we'll start releasing ETW executables linked with /LARGEADDRESSAWARE enabled, it's currently in test. This will at least save you the second part of the process (and having to repeat it!).
    I'm impressed you guys didn't have that much to correct in code, that you are at a testing stage. That is quite encouraging and I look forward to it. I applaud you for having a large address safe executable ready soon.

    Quote Originally Posted by JeromeGrasdyke View Post
    Just to expand a little further, the reason why this can cause both 'out of video memory' and 'out of system memory' issues is because DirectX grabs a fair amount of application address space for its buffers, and these resources are placed inside the user portion of the full 4 mb range.
    Thanks for the info, it adds to my understanding. It also may answer why all of the virt add space for the process didn't appear as accounted for by tools prior to crash, maybe because DX wrapped some of it in it's buffers separately as well, although not too sure of that really as some info on memory usage that people are having during crash comes at me second hand.

    Quote Originally Posted by JeromeGrasdyke View Post
    The memory issues are certainly not responsible for all problems people are reporting - for example crashes during the end-of-turn are down to other things, and there are a few other cases which can throw up identical errors - but it's one of several priority items for the programming team at CA right now.
    I was wondering about those end of turns issues too, as recently as tonight. Gladly I haven't experienced any of these yet. Kudos for responding and adding insight. +1 back at you for dev communication and adding insight.
    I appreciate you taking the time.
    Last edited by MadBoris; March 17, 2009 at 04:42 AM.

  20. #20
    Jango's Avatar Wimmer
    Join Date
    Jul 2007
    Posts
    162
    Tournaments Joined
    0
    Tournaments Won
    0

    Default Re: "Insufficient memory" & "CTD" - workaround

    Quote Originally Posted by JeromeGrasdyke View Post
    A very good summary of the issue MadBoris. At some point in the near future we'll start releasing ETW executables linked with /LARGEADDRESSAWARE enabled, it's currently in test. This will at least save you the second part of the process (and having to repeat it!).

    Just to expand a little further, the reason why this can cause both 'out of video memory' and 'out of system memory' issues is because DirectX grabs a fair amount of application address space for its buffers, and these resources are placed inside the user portion of the full 4 mb range. The memory issues are certainly not responsible for all problems people are reporting - for example crashes during the end-of-turn are down to other things, and there are a few other cases which can throw up identical errors - but it's one of several priority items for the programming team at CA right now.

    +1 rep
    This is excellent news!

Page 1 of 43 1 2 3 4 5 6 7 8 9 10 11 26 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •