TIP: App Pool High Memory Usage, intermittent site crashes

[AbleMods]

Ran into a situation today post-upgrade of Able 7.0.7 to Gold R6. Site was running great, but with intermittent instability. Throughout the day the instability grew more frequent.

We tried disabling Mobile, switching from SQL Full Text to SQL, even removing some Google Analytics code we installed earlier thinking it could be the cause. No luck.

App pool was confirmed set to ASP.NET 4.0/Integrated.

Server app pool CPU usage was low, like 5%-8% at best. But server app pool memory was running 800mb - 1,200mb . Restarting the app pool only helped briefly - within 30 seconds of restart we were back to the 800mb range and climbing. That's insane.

Eventually we checked one last thing: Advanced Settings on the app pool. Enable 32-bit Applications.

WHAM!! Memory immediately dropped to 250mb and holding steady. All we had to do was enable 32-bit apps in the app pool settings.

It's the little things that get 'ya

[NC Software]

Why is the memory at 800K to 1200K a concern? 64-bit apps can use more memory and that is a GOOD thing. You are forcing the store to use less memory as if you put the site into 32 bit mode you have process restrictions on the amount of memory used. Now if the site is showing signs of issues, then that could be a problem which goes back to server specs, how much RAM does the server have and is all of it available to the site? Our JIRA instance has memory usage over 2 GB but that's fine as it's in 64-bit mode and the server has plenty of memory. The days of the past of "KB" vs. MB/GB are changed but I'm curious as to what was really the issue with the site that you may be masking?

[AbleMods]

Because I know more about hosting AbleCommerce sites than you do.

[jguengerich]

Joe,
How about sharing some of your knowledge with the rest of the community by answering the question? Others may be curious too.

[AbleMods]

Any Able site that I've seen which exceeds 800mb RAM in the app pool will soon crash or at the very least grow increasingly unstable.

A low usage Able site runs 150mb-180mb RAM

A medium usage will push to 250mb-300mb

A raging Able site (>500 orders/day) will peak around 500mb RAM. In every single instance I've seen, the longer the app pool memory usage stays above 800mb, the faster the site stability deteriorates. I've never seen an Able site run longer than 60 seconds when the app pool consumes more than 1.2Gb of RAM.

Obviously customizations can play with the numbers. These numbers are assuming minimal, well written, thread-safe customizations.

Gold R5 had some issues with thread-safe caching of store settings. This caused excessive CPU and memory usage in high-traffic environments. This was fixed in R6. The issue did not manifest itself in low usage sites.

You can also explore deeper into your running .Net application using NewRelic - great tool for digging into live .Net apps.

[NC Software]

I do not agree with the above assessment or tip. It sounds like the hardware is insufficient to run the site but without specifying the details no one can comment. Setting 32 bit mode without full knowing the impact is unwise. If AC GOLD has a memory issue or instability issue it should be reported as a bug and allow the developers to comment or investigate. It could be flawed add-on code or any of hundreds of issues. You could try changing the session state from InProc to asp.net state server or even SQL server if potentially large objects in the cache are the cause. 800 process memory is not a concern on a properly equipped server and a large web application such as Able, JIRA, etc. may well exceed that. Do you think Amazon is or Google is running under 800 MB process space? I don't think so, don't expect an active site to be in this area. My AC 707 on a very powerful server, yet is a medium active site is just over 500 MB process usage. I expect GOLD to be higher than that due to the architecture and references.

Be very careful with taking anyone's advice without having all of the details or knowing exactly what you're doing. Again, I do not recommend this tip to fix a problem as outlined above, the software or add-ons itself may be flawed. You may as well take a "tip" to recycle the site every 24 hours which would be unwise as well.

Anyone that is running a GOLD site of any substantial activity I'd love to hear your assessment.

[NC Software]

What is the server OS:
How much RAM:
Processor type:
Processor quantity:
Hard drive type:
Hard drive RAID:
Is the site running under any type of virtualization:

[jguengerich]

Neal,

I don't have an active site yet (getting close ), so I can't give you any stats for my installation.

I do have a question. You mentioned recycling every 24 hours. The default Application Pool recycling setting is 24 hours. Are you suggesting that it is better to make that longer, or are you just saying that it is not necessary to do it manually?

[NC Software]

Jay,

I was joking along the line of recycling. I suggest leave IIS to defaults.

Moral of the story here is Joe has found an issue, while his fix may work for his situation, I don't suggest this tip is something everyone go out and do as there are significant ramifications of doing something like this. Instead I'd rather this situation be presented to AC for resolution stating that over time memory continues to rise and to the point that the site is crashing. There are a million variables, it could be data, it could be server, it could be other sites on the server, the list goes on. From Able's standpoint, it's hard to troubleshoot because once you start any customization or adding add-ons and then the solution provided by Able is no longer relevant.

I have been hesitant to use GOLD myself as 1) I have a great 707 site running now, and 2) GOLD has had a rough history. My planned entry point was and is R7 and I soon hope to start my build out but I have done extensive beta testing for Able as well as my own testing from pre GOLD to present. GOLD is currently trapped in legacy'ware by trying to support too old of servers and platforms which I've had a lot of conversation with them about. This concerns me because many of the references that GOLD even in R7 use are, in my opinion, out of date. This was my concern with Joe's post, is something not optimized in the GOLD package for 64 bit? There is talk that in R8 they will consider cutting ties with the old to bring the architecture more up to date but we shall see.

So long story longer, if there is a problem in GOLD R7 that Joe is pointing out, but I don't believe his scenario is using R7, then we better know now before even we at my company start our migration. I don't want to relive the StoreFront era of having to use asp.net state servers/services for terrible large object caching and performance issues, StoreFront has passed away, I don't want to see Able end up the same if this is a real problem. Hence, I'd love to hear from people with large sites in production, show us the web sites, how is the performance and memory issues, stability, etc. I do not know of many GOLD sites in production and would love to see them.

[jguengerich]

Neal,

I understand your concerns, but I can also understand Joe not wanting to spend a lot of time analyzing something when he has a simple fix that works for him.

I think you are correct that it may be difficult to pinpoint what is causing the issue. I guess a starting point would be someone (perhaps AbleCommerce) could set up an unmodified site and simulate frequent activity, then get some stats with and without the "Enable 32-bit applications" set. However, even if everything looked OK in that scenario, as you mentioned, there are multple SQL, IIS, Windows Server, etc. configuration scenarios that would have to be explored before you could have any certainty that the core AbleCommerce code was not to blame.

Our site will be low-usage for the forseeable future, and this is my first site (AbleCommerce or otherwise), so I probably can't contribute much in terms of real-world experience with a site with moderate to high activity.

I did manage to find this bug while developing our site: viewtopic.php?f=65&t=17828, but I have no idea if it is related to what we're discussing here.

[NC Software]

Jay,

I followed that thread and bug report - good job. That is the type of thing that concerns me that if there is an object like that in memory it could cause an issue. As you did, it needs to be raised to developers to fix properly. If an AC GOLD site cannot run months unattended, it is a problem.

I look forward to seeing your site when it's done and I'll share our new site/design with GOLD R7 or maybe R8 depending on how fast they push the next build.

[jguengerich]

Neal,

I work for a manufacturing company. Our site will be a "member only" site for our existing customers to order replacement and consumable parts. That means I won't be able to let the general public (including the AbleCommerce developer community) browse the site. Perhaps I could post some screenshots somewhere, but it will probably be a couple months before I do that. We hope to start using it in the next couple of weeks, but I'll probably be busy debugging things and catching up on some other projects before I get around to doing that.

[Katie]

I guess a starting point would be someone (perhaps AbleCommerce) could set up an unmodified site and simulate frequent activity, then get some stats

I just wanted to say that we actively doing just this. We're using a couple different tools to help us identify any bottlenecks or problems with Nhibernate when the store is under high traffic situations.

It is VERY difficult to troubleshoot issues like this, and I'm glad Joe found a solution that works for the time being. I will try to keep watch in Jira and see if anything major comes up.

[NC Software]

Katie / Devs

I have a site that will be dormant for a while until I get to doing the migration. If you all can provide me any suggestions or desires or how to stress test it then I'd be happy to do this for you.

[NC Software]

I have a GOLD R7 stock site attached to a restored/upgraded 707 database with the same content and config as http://www.nc-software.com. It's locked to my IP so no one can see it right now, but I hit the site, played around for 5 minutes discovering a bug with kitting sadly (reported) but in only a few minutes I'm at 775 MB. My 707 is at 650 MB as I write this. I'll look into putting together a test harness that continually hits pages on my site and see if the memory grows but it won't test any checkout paths and be able to stress that.

This server is a very powerful server running Windows Server 2008 R2 Standard, 32 GB RAM, Dual Proc (24 cores), SAS RAID 1 connected to SQL Server 2008 R2 Standard on another server, also very powerful, iSCSI SAN RAID-10 SAS. No virtualization.

GOLDR7.jpg

I suggest reviewing this:

http://help.ablecommerce.com/index.htm# ... ements.htm

Note:

* Due to a pending issue in Asp.Net 4.5, AbleCommerce Gold is not supported on a 32 bit server using medium trust.

Moral of the story, 64-bit applications WILL use more memory and should have sufficient RAM. I've seen server configs where people are running as low as 8 GB RAM on a server - this is ludicrous. I'd offer that you should have at least 16 GB RAM for 64-bit applications on a server such as GOLD, more is highly recommended. 800 MB is nothing to worry about and I would suggest looking elsewhere for the issue raised above and also try out of process state server such as the asp.net state service as the resolution. If you have a server inadequate to handle e-commerce and GOLD then you will end up with poor results. We have yet to learn the configuration, for whatever reason, of Joe's scenario.

[NC Software]

I'm going to give selenium a try: Open with FireFox - http://docs.seleniumhq.org/

[NC Software]

Added a .js file to loop selenium - those that want to use this here is the link: http://stackoverflow.com/questions/1103 ... lenium-ide

I'm running a continuous loop on my acdev site now, it's essentially single user continually hitting the site adding products to the cart, clearing the cart, going through kits, products with options, searches, etc. I'll watch it over the next 24+ hours and we'll see how memory goes. Here is where the memory is now after 5 minutes of the macro running:

TestStart.jpg

[NC Software]

About 2.5-3 hours in to the test, did a check and memory is stable at around 850 MB, little processor activity, site continues to perform well, even hitting it from a second computer and site is doing great.

acdev-2hrs.jpg

Again, stock/new GOLD R7 no customizations, two server config web & sql, no problems noted. Memory is right where it is expected, AC 707 is slightly lower as expected as well.

[NC Software]

Overnight - memory stable, site performant, test terminated.

I'm using SQL FTS, low values for maintenance storage (anonymous, page tracking, etc), and I'm not using google feed while in development.

[Katie]

Very nice! Thank you for your efforts Neal