Server performance issues

For general questions and discussions specific to the AbleCommerce 7.0 Asp.Net product.
Post Reply
User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Server performance issues

Post by WylieE » Tue Jan 20, 2009 3:05 pm

OK, so we attempted a rollout of our new AC site today. Unfortunately, it did not go so well. The server CPU choked under ASP load. Memory and disk activity were fine. We're digging through our stats to see if something went wrong and could use some feedback.

We had the site live for approximately 40 minutes. During that time, we had almost 600 unique visitors. Average outgoing bandwidth is about 2.6M/s. As far as overall traffice goes, we normally average about 9k in page hits a day. (689gig/mo)

I'm presuming we may be a bit above average for AC sites in terms of traffic? Do we need to upgrade our server or should it be able to handle the load better?

Server specs:
P4 2.8ghz, 512kb cache
2gig RAM
SCSI drives
Win Server 2003
.Net 2.0

AbleCommerce install
PLATFORM: ASP.NET
VERSION: 7.0
BUILD: 10863

AjaxControlToolkit: 1.0.10301.0
AxInterop.ZRush_ShipRush5: 1.0.0.0
CommerceBuilder: 7.0.10752.0
CommerceBuilder.AuthorizeNet: 7.0.9764.0
CommerceBuilder.CanadaPost: 7.0.0.0
CommerceBuilder.CCH: 7.0.0.0
CommerceBuilder.Configuration: 7.0.10752.0
CommerceBuilder.CyberSource: 7.0.9764.0
CommerceBuilder.DataClient.Api: 1.0.0.21
CommerceBuilder.DataClient.Csv: 1.0.0.0
CommerceBuilder.DHL: 7.0.8825.0
CommerceBuilder.ECB: 7.0.0.0
CommerceBuilder.FedEx: 7.0.9725.0
CommerceBuilder.FRBNY: 7.0.0.0
CommerceBuilder.GoogleCheckout: 7.0.10052.0
CommerceBuilder.InternetSecure: 7.0.9764.0
CommerceBuilder.LinkPoint: 7.0.9764.0
CommerceBuilder.Paradata: 7.0.9764.0
CommerceBuilder.PayFlowPro: 7.0.10061.0
CommerceBuilder.PayJunction: 7.0.9764.0
CommerceBuilder.PayPal: 7.0.10840.0
CommerceBuilder.Services: 7.0.10752.0
CommerceBuilder.SkipJack: 7.0.9764.0
CommerceBuilder.UPS: 7.0.10041.0
CommerceBuilder.USPS: 7.0.9728.0
CommerceBuilder.Web: 7.0.10752.0
ComponentArt.Web.UI: 2007.1.1566.3
ConfiguredItems: 1.0.0.0
CyberSource.Base: 2.0.0.0
CyberSource.Clients: 5.0.2.0
CyberSource.Clients.XmlSerializers: 5.0.2.0
CyberSource.WSSecurity: 2.0.0.0
CybsWSSecurityIOP: 1.0.0.0
DevExpress.Data.v6.3: 6.3.3.0
DevExpress.Utils.v6.3: 6.3.3.0
DevExpress.XtraBars.v6.3: 6.3.3.0
DevExpress.XtraEditors.v6.3: 6.3.3.0
DevExpress.XtraGrid.v6.3: 6.3.3.0
edtFTPnet: 1.2.4.0
FredCK.FCKeditorV2: 2.5.2912.21007
ICSharpCode.SharpZipLib: 0.85.5.452
Interop.ZRush_ShipRush5: 1.0.0.0
LinkPointTransaction: 1.0.1797.29766
log4net: 1.2.10.0
MAX.USPS: 1.0.0.0
Microsoft.Practices.EnterpriseLibrary.Common: 2.0.0.0
Microsoft.Practices.EnterpriseLibrary.Data: 2.0.0.0
Microsoft.Practices.ObjectBuilder: 1.0.51205.0
Microsoft.Web.Preview: 1.2.61025.0
NVelocity: 7.0.8530.0
PaygatewayNET: 3.0.0.0
paypal_base: 4.2.1.0
PFProCOMLib: 1.0.0.0
PFProdotNET: 0.0.0.0
skmMenu: 2.2.2991.25093
UPSOnlineInterface: 1.0.0.0
UsaEpay: 1.0.0.0
USAePayAPI: 1.5.2818.15473
Validators: 1.0.0.0
WarmothBody: 1.0.0.0
WebChart: 1.1.1.6
wwHoverPanel: 1.85.0.0
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

kastnerd
Commodore (COMO)
Commodore (COMO)
Posts: 474
Joined: Wed Oct 22, 2008 9:17 am

Re: Server performance issues

Post by kastnerd » Tue Jan 20, 2009 3:35 pm

how large is your database?

User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Re: Server performance issues

Post by WylieE » Tue Jan 20, 2009 4:16 pm

kastnerd wrote:how large is your database?
The AC database itself is only a bit over 10mb.
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

Mike718NY
Commodore (COMO)
Commodore (COMO)
Posts: 485
Joined: Wed Jun 18, 2008 5:24 pm

Re: Server performance issues

Post by Mike718NY » Tue Jan 20, 2009 4:51 pm

Is the database on it's own server? I think it needs to be.

User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Re: Server performance issues

Post by WylieE » Tue Jan 20, 2009 5:31 pm

Mike718NY wrote:Is the database on it's own server? I think it needs to be.
I don't see that as a system requirement, but SQL is on its own server. Quad core Xeon processor, mucho RAM, running SQL server 9.0.3042. I don't have the specs handy but it is a much newer and more powerful box.

When we attempted the rollout today, the SQL server seemed to be cruising along with no problems. The web server was completely choking.

Any recommendations on stress testing tools for IIS? We don't have enough staff to come close to hundreds of customers in a short period like our live site gets. Prior to the rollout, we had used our staff to create test orders and bounce around the site. (@ 20 employees) Worked fine for the smaller load.

Thanks.
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

User avatar
AbleMods
Master Yoda
Master Yoda
Posts: 5170
Joined: Wed Sep 26, 2007 5:47 am
Location: Fort Myers, Florida USA

Re: Server performance issues

Post by AbleMods » Tue Jan 20, 2009 7:18 pm

Whole lot of things could be coming into play.

690 visitors in 40 minutes isn't small but it's not gigantic either. The first question(s) I would ask would be:

1. Look at processes on the server via Task Manager and see what specifically is hogging CPU.
2. Were you uploading any programming changes to the site during this time? Or making changes to the web.config or ~/Bin/ folder? If so, recompiles will bog it down badly, especially on a single-core P4 with only 512kb of cache.
3. Speaking of which, my memory of P4 chipsets recall 2.8Ghz single-core models having 1Mb of cache. You sure the server isn't a Celeron?
4. Given the CPU seems to be the hog, doesn't sound like a bandwidth issue. http://www.paessler.com/prtg has a nice utility that'll let you SNMP the bandwidth figures right off the server NIC and anything else that supports SNMP. Might be useful to chart bandwidth in case you're getting DoS slammed.
5. Maybe some more information on exactly HOW you transitioned to live might lend some clues. Did you create a bunch of 301 redirects? Did you run the IIS lockdown tool prior to enabling the site? Was it just a simple DNS change?
6. What sort of other processes are you running on the web server? Anti-virus? Anti-spyware? Spybot? Ad-aware?
7. Anything scheduled in task scheduler?
8. I notice you have ShipRush installed. Does it have any automated or startup processes that could be causing the issue under load? Is there a way to bring the site live without it just to eliminate a possible variable and get the site closer to AC7 standard?
9. Windows Firewall enabled? IP Filtering enabled?
10. After you transitioned to live, can you shut down all inbound traffic and just hit the site from the server console. Does the CPU still spike?

That's all I can think of at the moment, sorry for the long list.
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com

User avatar
AbleMods
Master Yoda
Master Yoda
Posts: 5170
Joined: Wed Sep 26, 2007 5:47 am
Location: Fort Myers, Florida USA

Re: Server performance issues

Post by AbleMods » Tue Jan 20, 2009 7:19 pm

11. Can you confirm if your bandwidth between the web server and the SQL server is being throttled or not?
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com

User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Re: Server performance issues

Post by WylieE » Wed Jan 21, 2009 3:22 pm

SolunarServices wrote:Whole lot of things could be coming into play.

<... stuff snipped...>

That's all I can think of at the moment, sorry for the long list.
Thank you Joe!

Unfortunately, our server is offsite and I do not have direct access to it. The hosting company handles all the server management for us. I'm trying to get some sort of login so I can closely monitor activity and configuration.

Our live server also hosts our current website and eStore solution. Performance of our current site (php/mysql based) has always been good. When we attempted the rollout, our hosting company switched the domains around (DNS changes).

We've found the MSFT Web Application Stress program and are testing it on our internal development server. What we've found so far is the nav bar application we were using is terribly inefficient. Removing it significantly speeds up performance. We have an alternate solution we're testing now and response so far has been good.

I'll post back as I have more specifics.
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

Mike718NY
Commodore (COMO)
Commodore (COMO)
Posts: 485
Joined: Wed Jun 18, 2008 5:24 pm

Re: Server performance issues

Post by Mike718NY » Wed Jan 21, 2009 4:32 pm

Are you on "shared hosting"? If so, that could very well be the problem.
I was originally but had to change hosting companies because of poor performance. I moved to another company that had better performance, until December. Apparently there was one website on that server that was getting a lot of hits and slowing down the performance of other site, like mine.
So I had the website moved to a more isolated enviroment (Private MS SQL Server Database, Private Virtual IP Block, Private Windows Server 2008 Instance).
Now, there should not be any performance issues caused by other sites.
But it's cost $170.00 a month now.

See if your hosting company offers a better performing package. If you are paying $30 or $40 a month, you'll probably need to pay a lot more.

User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Re: Server performance issues

Post by WylieE » Wed Jan 21, 2009 4:36 pm

Actually, we own the servers. We do not share the servers with anyone else. The hosting company handles all the ongoing maintance of the boxes.
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

User avatar
AbleMods
Master Yoda
Master Yoda
Posts: 5170
Joined: Wed Sep 26, 2007 5:47 am
Location: Fort Myers, Florida USA

Re: Server performance issues

Post by AbleMods » Wed Jan 21, 2009 4:50 pm

I believe I just got snipped :P

I'll be curious to learn what exactly about the Nav bar was causing the issue should it turn out to be the cause. Good luck!
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com

User avatar
BryanWarmoth
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 44
Joined: Fri May 23, 2008 11:24 am
Location: Puyallup, Wa
Contact:

Re: Server performance issues

Post by BryanWarmoth » Thu Jan 22, 2009 6:22 pm

We used the same nav bar that is in the admin section of Able Commerce. Our nav bar had about 400 nodes and I'm guessing that it doesn't scale very well because as soon as we replaced it our site started running very quick. I was wondering if anyone else has tried to use this same nav bar outside of the admin area. Even when we have a couple people using the admin area we see huge spikes in our CPU every page load. Are you guys seeing these same spikes when you use the admin area? We're trying to figure out if the nav bar is really slow or if it is some setting that we are missing? I realize that 400 nodes is a lot for a nav bar but I would think the small amount of nodes in the admin area wouldn't cause our CPU to spike. Any ideas?
Bryan Bingham
Warmoth Guitar Products Inc.
bryan@warmoth.com
http://www.warmoth.com

User avatar
AbleMods
Master Yoda
Master Yoda
Posts: 5170
Joined: Wed Sep 26, 2007 5:47 am
Location: Fort Myers, Florida USA

Re: Server performance issues

Post by AbleMods » Thu Jan 22, 2009 11:33 pm

No nav-bar is going to handle 400 nodes well. Your page size must be massive. SEO will get killed - bots don't like massive pages flooded with links.

On the other hand, the existing menu on the admin side should not result in significant CPU spikes on the server.

ComponentArt uses AJAX. You might want to look at reinstalling the AJAX extensions on the server in case something has caused them to corrupt. Would be the first time I've ever heard of it but I'm by no means an authority.

Something definitely isn't as it should be - my server hums at 99% idle bouncing around the Admin menu pages using the drop down menus. And it's a G3 Proliant dual Xeon 3.06Ghz so it's not exactly the latest.
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com

pizzaovens
Ensign (ENS)
Ensign (ENS)
Posts: 19
Joined: Wed Oct 01, 2008 10:05 am

Re: Server performance issues

Post by pizzaovens » Mon Jan 26, 2009 4:05 pm

we are running our site with sql hosted on the same server and are experiencing frequent SQL deadlock erros on product.aspx that slow the site down to a crawl for 5-10 minutes. could this be because the sql dadabase is local to the web drive ?

User avatar
AbleMods
Master Yoda
Master Yoda
Posts: 5170
Joined: Wed Sep 26, 2007 5:47 am
Location: Fort Myers, Florida USA

Re: Server performance issues

Post by AbleMods » Mon Jan 26, 2009 4:50 pm

pizzaovens wrote:we are running our site with sql hosted on the same server and are experiencing frequent SQL deadlock erros on product.aspx that slow the site down to a crawl for 5-10 minutes. could this be because the sql dadabase is local to the web drive ?
Not that I know of. I run my SQL db locally as well and never see deadlock issues or errors. Something else must be the cause.

What causes a SQL deadlock error?
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com

User avatar
jmestep
AbleCommerce Angel
Posts: 8164
Joined: Sun Feb 29, 2004 8:04 pm
Location: Dayton, OH
Contact:

Re: Server performance issues

Post by jmestep » Mon Jan 26, 2009 5:43 pm

In the past before some performance mods Able came up with for 5.5 (I'm sure they have included these in 7 already), we had deadlocks when a spider like Yahoo hit the site, accessing hundreds of links at the same time. Each spider was trying to get a user id and it clogged the connection to the SQL server, so it could be something similar to that- some kind of latency problem.
Judy Estep
Web Developer
jestep@web2market.com
http://www.web2market.com
708-653-3100 x209
New search report plugin for business intelligence:
http://www.web2market.com/Search-Report ... -P154.aspx

pizzaovens
Ensign (ENS)
Ensign (ENS)
Posts: 19
Joined: Wed Oct 01, 2008 10:05 am

Re: Server performance issues

Post by pizzaovens » Tue Jan 27, 2009 1:19 pm

jmestep wrote:In the past before some performance mods Able came up with for 5.5 (I'm sure they have included these in 7 already), we had deadlocks when a spider like Yahoo hit the site, accessing hundreds of links at the same time. Each spider was trying to get a user id and it clogged the connection to the SQL server, so it could be something similar to that- some kind of latency problem.
any way to fix this ?

User avatar
WylieE
Captain (CAPT)
Captain (CAPT)
Posts: 281
Joined: Tue Mar 25, 2008 8:26 am
Location: Puyallup, WA
Contact:

Re: Server performance issues

Post by WylieE » Wed Jan 28, 2009 11:31 am

WylieE wrote:
Mike718NY wrote:Is the database on it's own server? I think it needs to be.
I don't see that as a system requirement, but SQL is on its own server. Quad core Xeon processor, mucho RAM, running SQL server 9.0.3042. I don't have the specs handy but it is a much newer and more powerful box.

When we attempted the rollout today, the SQL server seemed to be cruising along with no problems. The web server was completely choking.

Any recommendations on stress testing tools for IIS? We don't have enough staff to come close to hundreds of customers in a short period like our live site gets. Prior to the rollout, we had used our staff to create test orders and bounce around the site. (@ 20 employees) Worked fine for the smaller load.

Thanks.
Has anyone used the Microsoft Web Application Stress Tool, WCAT, WebLoad or vPerformer for stress testing your website? We can run the same script steps and user loads and get wildly different results. It is difficult to get a reasonable estimate as to how many users our new site can handle, let alone help determine what is causing our intermittent slow response times.

Much of the time, we're seeing 3 -5 second page load times. Intermittent slowdowns are 20 - 45 seconds for page load. Our current site averages 0.3 - 1.0 second page load. The difference between sites is like night and day for our customer base. We need to get the site more consistent and streamlined. Even though the new site is much more full featured, we'll lose customers if we cannot speed it up.

AC Staff: Do you have any benchmark tests we could run and/or compare against our site? We are also seeing the slow page loads in the admin section. Please? Pretty please with Starbucks on top??

Thanks,
Eric Wylie
Warmoth Guitar Products, Inc.
http://www.warmoth.com

Post Reply