Site went down this morning twice with weird error.

[LauraNichols]

This morning I woke up to my site down. The custom error page did not show up and the white and yellow .NET screen came up showing this error. I had to restart IIS to fix it. I'm running AC 7.0.7 and just two days ago I recompiled the source code using VS2010 and .NET 4.0 in release mode and pushed it up to the server. It's been running fine for two days, so not sure if that had something to do with it or not as this seems really random. I'm running Windows 2008, IIS7 Integrated mode. Site's been running for 2 months now with no serious issues until now. Any ideas?

Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at System.Web.PipelineModuleStepContainer.GetEventCount(RequestNotification notification, Boolean isPostEvent)
at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)

[LauraNichols]

I just noticed that right before the site goes down, there are several errors in the windows log saying

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at CommerceBuilder.Data.Database.OpenConnection()
at CommerceBuilder.Data.Database.ExecuteScalar(DbCommand command)
at CommerceBuilder.Stores.BannedIPDataSource.IsBanned(String ip)
at CommerceBuilder.Services.c.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[dgoranov]

The exception is caused by mssql database connection timeout or overloaded mssql database server. Are you using a private dedicated mssql
database server instance or the database is running in a shared hosting environment?

[AbleMods]

I had a similar problem with my server for a while. Was always in the mornings too, never afternoon or evenings.

Figured out I had a couple of different automatic AV and security update routines firing off within a few minutes of each other around 5:00am. This caused the SQL box to slow down so badly that my IIS/.Net would time-out trying to talk to it.

Once IIS/.Net encountered the timeout, it would never talk to the SQL box again until I restarted IIS. Never figured out why the framework wouldn't talk to SQL any more. But changing the starting time for the update routines resolved it immediately.

[LauraNichols]

We were running on the same dedicated box, which was great for the amount of traffic we had, but we've been doing some promoting and our traffic has recently increased, so after much talk and debate, last night we moved our Sql Server over to it's own box, and I'm still seeing timeout errors. A few weeks ago I had to increase the max pool size to 400 in the connection string and that helped, but I still see timeout errors.

I'll check out the sercurity update routines also and make sure something isn't running when it shouldn't be, but I don't think that's it. I get these errors all times of the day and sometimes I can go days without an error. I think this is traffic related, but can't figure out if it's an issue with the Sql box needing more memory, or a connection pool issue.

How much traffic should AC be able to handle? I would think a ton. Right now we're getting up to 10k hits per day, but sometimes I see timeouts on slow days when we're only getting 2500 hits.

[AbleMods]

How much traffic should AC be able to handle? I would think a ton. Right now we're getting up to 10k hits per day, but sometimes I see timeouts on slow days when we're only getting 2500 hits.

My Solunar.com site currently has an average-hits-per-day of 48,000+. So AC7 can certainly larger volumes of traffic.

If you're getting timeouts on 2,500 hits per day, something is definitely still wrong. But it's not going to be easy to trace down. Usually a good DBA with direct access to both boxes can use SQL Profiler to see what's causing the bottlenecks.

Some areas to check:
1. If there's a firewall between web server and SQL server, make sure it doesn't throttle traffic.
2. Make sure both servers have latest NIC drivers. I've seen older NIC drivers fail under load.
3. Check the Processes list, how's the CPU utilization? If you're running just 1 site on that server, should be less than 10% given your traffic. If something's hogging the CPU, figure out what process it is and shut it down for a while to see if things improve.
4. Any storefront customizations? Poorly written SQL queries buried in custom code could bring the entire site to it's knees. I'm working with a client now with that exact problem. Very time-consuming to trace down but we figured it out.

Overall it's a tough road to walk...troubleshooting stuff like this is very trial-and-error sometimes.

[LauraNichols]

I just wanted to follow up here for future readers that I found the issue with this. I had an error in how I was instantiating the connection in my custom code, so it was leaving connections open. When the max pool size was reached it would crash. As soon as I fixed the error, the connection pool stopped growing and the site has not had this error again. I was able to find this by monitoring the current connections in the performance monitor on the webserver.