Server Reboot - Have to manually restart application pool

[gdelorey@mitcs.com]

Hello -

We are managing an AbleCommerce 7.0.7 site for a client (we started on AC 5.5) and I've noticed in the past few months that every time we do a server reboot and startup, the site remains offline with an asp.net error:

Event message: An unhandled exception has occurred.

I have to manually go into the site to restart the application pool, then it comes up fine. This is proving to be a hassle since Rackspace installs Windows updates monthly and then I have to try and restart the app pool early in the morning before customers are affected. Anyone else run into this issue or have an idea of what's going on?

Thanks,
Greg

[LauraNichols]

I have! If you can go into the windows logs and check out the complete error message it will help to see if it's the same issue I was having, but what was happening to me was that our database server was restarted due to windows updates, and someone hit the site, the site would not be able to log in to the database because the server was down. This messes up the Token.Instance.User, which is hit first in AdminPanel.ascx.cs (which you would see in the error log) and makes it null, and it totally hoses the site and the only way to fix it is an app pool recycle.

I couldn't figure out a fix in the code to solve this, I wish I could, but Token.Instance is getting set in a dll that I don't have source code for (and I purchased the source code, but there are still dll's that aren't included), but what I did do was not allow automatic updates. I also have the application pool scheduled to restart every 24 hours in the middle of the night. If you can't get rid of the windows updates (which is what I would do if you can), make sure your application pool on your webserver gets restarted after the reboot. I had some success with this, but sometimes I'd have to go in and restart IIS completely.

Also --- check to make sure you have Auto_Close on your database set to False, this is really important and I think was part of the issue causing my problems too.

I actually have just figured this out, as my site was going down once a week for three weeks in a row, I'm 10 days in now with no errors, so keeping my fingers crossed that I've bandaided the issue. I wish I had a complete fix, something in the code would be nice. Probably need to put in a ticket with AbleCommerce and let them know it's an issue. I read somewhere that this was supposed to be fixed in 7.0.5, but I'm seeing the issue in 7.0.7, which is what I am running.

[gdelorey@mitcs.com]

Hi Laura -

That sounds exactly like the same issue - I'll look into submitting a bug report to AC on this.

Thanks,
Greg

[sloDavid]

I just experienced this wonderful error for the first time this morning. Was the bug submitted to AC?

[LauraNichols]

I have not submitted. I don't know how. Where do you go to submit bugs?

[sweeperq]

We've had the same issue a few times one our AC7 sites. Only way to get the site back up is an application recycle

[LauraNichols]

I went to go report the bug and clicked on the link to report a bug from their support page and the page has been removed, so I can't report it. Maybe they don't want to hear about their bugs anymore.

[sweeperq]

I usually submit bug reports through my account page. When you submit the report, make sure to point to any threads in the forums that show other people are having the same issue.

----

Just checked with Fred on chat. Gave me this help doc: http://help.ablecommerce.com/faqs/ablec ... a_bug_.htm. Looks like they are starting to lean to a paid support model and not charging if it turns out to be a bug If you go ahead, definitely point back to the threads to show it is happening for multiple customers.

[gdelorey@mitcs.com]

I just submitted a bug report. They don't have the link any longer - you have to email them at bugs@ablecommerce.com.

Greg

[Katie]

Thanks for making us aware of the issue. I'm still not certain if the developers would consider this a bug yet. The main reason I'm hesitant to say for certain it's a bug, is that we manage several servers and have many installations of AC7 running. As far as I know, we haven't had any issues that are consistently happening. Of course, from time to time, a server needs to be rebooted or the app pool needs recycled - but it's rare.

If the issue only happens when automatic updates are turned on, then maybe a quick test would be to turn this off as Laura mentioned in her post -

but what I did do was not allow automatic updates. I also have the application pool scheduled to restart every 24 hours in the middle of the night. If you can't get rid of the windows updates (which is what I would do if you can), make sure your application pool on your webserver gets restarted after the reboot. I had some success with this, but sometimes I'd have to go in and restart IIS completely.

Also --- check to make sure you have Auto_Close on your database set to False, this is really important and I think was part of the issue causing my problems too.

So, if you're having this problem right now, please try the suggestions above and report back. Please list the version of AC7 you are using as well.
In the meantime, I'll see what else I can find out about the issue.

Katie

[Shopping Cart Admin]

Hello,

The issue corrected in patch 7.0.5 didn't have anything to do with restarting the web server, only the database server. I just confirmed with hosting that we've seen no issues with re-starting the web server, regardless of version on many hundreds of re-starts over the years.

8646 normal API IIS Application pool stops when database connection is unavailable

I'm doubtful there is something code related we can do from my reading of several posts I found on google. However please be specific on what is being re-started (webserver or dbserver), what version of Windows your running and what version of AbleCommerce your running. Then if there is some common theme it gives us a foundation to work from.

[gdelorey@mitcs.com]

Since our web & db servers are normally rebooted around the same time, I can't say for sure but I'm guessing it's the DB server reboot that causes the issue. Our DB server is Windows 2008 with SQL Server 2008 Web Edition (not R2). We are running AC 7.0.7.14481.

Greg

[LauraNichols]

Mine was the db server. What was happening was when the db server rebooted, AbleCommerce couldn't log in, and then Token.Instance.User would become null. This would literally take down our site until we recycled the app pool. Because this was happening in the middle of the night our site would be down for hours before we knew about it. I fixed the issue with the automatic updates rebooting the server, and it still happened. I then realized I had auto_close set to true, and ever since I changed that, I haven't had an issue.

The site should be able to recover from the db server not being available, so I do consider this not only a bug, but a pretty big one.

Please see this thread that I started when I had the issue the first time.
viewtopic.php?f=42&t=16203

[LauraNichols]

Also, I'm running AbleCommerce 7.0.7, Sql Server 2008 SP1, Windows 2008 R2 64 bit.

[Shopping Cart Admin]

Hello

The site should be able to recover from the db server not being available, so I do consider this not only a bug, but a pretty big one.

This issue was corrected in 7.0.5, so it's NOT the db issue as that was already solved. In further searching it appears the 2008 doesn't always restart the app pool as it should.

From my reading on this subject it would take some kind of helper application that monitors the health of the app pool independently of AbleCommerce.

While I might be wrong and I have been in the past, I'm pretty certain there isn't anything within the code of AbleCommerce that we could change to correct this issue. I'm looking forward to getting a few more updates from others to see if it narrows down the issue more.

[jmestep]

We have the issue on sites using SQL Server 2008R2 and I snagged some event logs from my desktop computer with SQL Server 2008R2- same computer as the IIS for the sites is. I have been having trouble recently with SQL Server not restarting after I do windows updates and restart my computer (I've got it set to auto start, restart on each failure but it doesn't start), but I hadn't thought to look for events logged before. Once I got the database restarted, I had to refresh the Able page on a 7.0.7 sr1a build a couple of times to get the site to start. (I talked to a guy at Microsoft a couple of days ago using some advisory hours I get with my Web developer and designer action pak subscription and he said it wasn't a known issue to him, but if I got him some error logs, he could check on it/refer me to someone else)There is nothing in the Able app.log, but here are some error messages from the Windows event log. The errors are not actually from the page itself, but from a Googlebase feed I have set to run on schedule in the global.asax file.
(These are in reverse order time-wise.)

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: System.Data
P5: 4.0.0.0
P6: 4dd244d5
P7: 2797
P8: 3
P9: System.Data.SqlClient.Sql
P10:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
Exception code: 0xe0434352
Fault offset: 0x000000000000cacd
Faulting process id: 0xbc4
Faulting application start time: 0x01cd495974e95e18
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: da161761-b54c-11e1-a390-00221910717c

Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Data.SqlClient.SqlException
Stack:
at CommerceBuilder.Data.Database.OpenConnection()
at CommerceBuilder.Data.Database.ExecuteReader(System.Data.Common.DbCommand)
at CommerceBuilder.Stores.Store.Load(Int32)
at CommerceBuilder.Stores.StoreDataSource.Load(Int32, Boolean)
at ASP.AcApp.TimedEvent(System.Object)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._TimerCallback.PerformTimerCallback(System.Object)

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1/ROOT/Able14600Test2

Process ID: 3012

Exception: System.Data.SqlClient.SqlException

Message: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

StackTrace: at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at CommerceBuilder.Data.Database.OpenConnection()
at CommerceBuilder.Data.Database.ExecuteReader(DbCommand command)
at CommerceBuilder.Stores.Store.Load(Int32 storeId)
at CommerceBuilder.Stores.StoreDataSource.Load(Int32 storeId, Boolean useCache)
at ASP.AcApp.TimedEvent(Object stateInfo) in e:\Able14600Test2\Global.asax:line 101
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading._TimerCallback.PerformTimerCallback(Object state)

[dgoranov]

Hi Judy,

We have been running AbleCommerce 7.x on SQL Server 2008R2 x64 for years and have not had any issues. According to the error log the application pool crashes because
of database connection issues. If the mssql server instance is up and running and you can connect to the database server using SQL Management studio with with the same
user/password set in the database.config file. The database user set in the database.config file must own the database schema if the schema is different from dbo and must
have db_owner set on the AbleCommerce database instance.

If the database connection is trough " Named Pipes Provider" no TCP/IP the connection string should have the following format:

connectionString="Server=DEVEL-PC\MSSQL;Database=ablecommerce-ac7;Uid=devel;Pwd=devel;"

[sloDavid]

What if we reset the Token Instance? When Application_Error handles an exception, at the end, after logging the exception, we could add

Code: Select all

if (exception is System.Data.SqlClient.SqlException) Token.ResetInstance();

Or, from a more symptomatic perspective,

Code: Select all

if (Token.Instance == null || Token.Instance.User == null) Token.ResetInstance();

Does anybody know if that would work? Or is at least safe to do? Maybe that could cause a problem if ResetInstance() itself encounters a problem, and we should instead do it on Application_BeginRequest()?

[jmestep]

The SQL server had stopped and I had to restart it. Then after accessing the page anew, I was able to access the site. I don't know what errors they are getting on our servers, but they are supposed to let me see them next time they do windows updates.
It will be interesting, since you are not seeing the same behavior with a similar setup.

[sloDavid]

The site should be able to recover from the db server not being available, so I do consider this not only a bug, but a pretty big one.

This issue was corrected in 7.0.5, so it's NOT the db issue as that was already solved. In further searching it appears the 2008 doesn't always restart the app pool as it should.

It happens to me on a clean-install 7.0.6, but never occurs on a 7.0.4 install that I have running, so maybe there was another issue.

The actual line of code that fails occasionally is in /Layouts/AdminPanel.ascx.cs:

Code: Select all

if (Token.Instance.User.IsInRole(Role.WebsiteAdminRoles))

Page_Init on that file seems to be the earliest event that we can tap into where Instance.User ought to be nonnull. So this morning, I'm adding an additional if-clause immediately before that code:

Code: Select all

if (Token.Instance == null || Token.Instance.User == null) {
  Token.ResetInstance();
  Response.Clear();
  Response.ClearHeaders();
  Response.ClearContent();
  Response.Write("Please try reloading the page. We apologize for the inconvenience.");
  Response.End();
}

I don't know what ResetInstance() even does. We'll see if the problem goes away...

[jmestep]

I had this problem over the weekend. I opened a dev site on my desktop and SQL Server had stopped. This is the error message.
The line 64 in the global.asax is where it logs the exception in the Application_Error method- Logger.Error(errorInfo, exception);

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 6/17/2012 9:14:13 AM
Event time (UTC): 6/17/2012 1:14:13 PM
Event ID: 3fa7787abee241ec92c27faa8365688c
Event sequence: 2
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/1/ROOT/Able14600Test6-1-129844124311174997
Trust level: Full
Application Virtual Path: /Able14600Test6
Application Path: E:\Able14600Test6\
Machine name: PRECISION

Process information:
Process ID: 7136
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE

Exception information:
Exception type: SqlException
Exception message: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

Request information:
Request URL: http://localhost/Able14600Test6/login.aspx?ReturnUrl=An unhandled exception has occurred.fAble14600Test6An unhandled exception has occurred.fadminAn unhandled exception has occurred.fdefault.aspx
Request path: /Able14600Test6/login.aspx
User host address: ::1
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\NETWORK SERVICE

Thread information:
Thread ID: 6
Thread account name: NT AUTHORITY\NETWORK SERVICE
Is impersonating: False
Stack trace: at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at CommerceBuilder.Data.Database.OpenConnection()
at CommerceBuilder.Data.Database.ExecuteScalar(DbCommand command)
at CommerceBuilder.Configuration.TokenBase.a(Database A_0)
at CommerceBuilder.Configuration.TokenBase.a(HttpApplicationState A_0, HttpRequest A_1, AbleCommerceApplicationSection A_2, Database A_3)
at CommerceBuilder.Configuration.TokenBase..ctor()
at CommerceBuilder.Common.Token..ctor()
at CommerceBuilder.Common.Token.get_Instance()
at CommerceBuilder.Utility.ErrorMessage.Save()
at CommerceBuilder.Utility.Logger.a(String A_0, LogMessageType A_1, Exception A_2, Type A_3)
at ASP.AcApp.Application_Error(Object sender, EventArgs e) in e:\Able14600Test6\Global.asax:line 64
at System.EventHandler.Invoke(Object sender, EventArgs e)
at System.Web.HttpApplication.RaiseOnError()

I restarted SQL server and refreshed the admin page- same error. I checked SSMS and database was available.
I recycled the app pool a couple of times, restarted the website- same error.
Then when I did an IISReset and then refreshed the page, the site came up OK.
I do not have ping enabled in IIS because of not wanting to get bumped out of debugging in visual studio if I'm debugging a process I'm stepping thru.

[napacabs]

David, did the extra code fix the problem?

The site should be able to recover from the db server not being available, so I do consider this not only a bug, but a pretty big one.

This issue was corrected in 7.0.5, so it's NOT the db issue as that was already solved. In further searching it appears the 2008 doesn't always restart the app pool as it should.

It happens to me on a clean-install 7.0.6, but never occurs on a 7.0.4 install that I have running, so maybe there was another issue.

The actual line of code that fails occasionally is in /Layouts/AdminPanel.ascx.cs:

Code: Select all

if (Token.Instance.User.IsInRole(Role.WebsiteAdminRoles))

Page_Init on that file seems to be the earliest event that we can tap into where Instance.User ought to be nonnull. So this morning, I'm adding an additional if-clause immediately before that code:

Code: Select all

if (Token.Instance == null || Token.Instance.User == null) {
  Token.ResetInstance();
  Response.Clear();
  Response.ClearHeaders();
  Response.ClearContent();
  Response.Write("Please try reloading the page. We apologize for the inconvenience.");
  Response.End();
}

I don't know what ResetInstance() even does. We'll see if the problem goes away...

[sloDavid]

Tom,
We haven't had any further troubles, but we also haven't had any more instances where the database login failed, so I don't know for certain. I also still haven't heard from AC or anybody else what the ResetInstance method does. If you implement it, can you let us know how it goes?

[stewie1570]

We've been experiencing this problem as well. I can reproduce the issue in a localhost environment by simply stopping the SQL service and sending many requests over a period of about 2 minutes while the db is down. The first error that i get is this one:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

On occasion I do notice a null reference exception on the Token.Instance.User (Instance is null) coming from /Layouts/AdminPanel.ascx.cs.. After a few seconds of sending requests while the db is down (to mimic traffic to the site while the db is down) I notice the error message change to this (I included the call stack as well):


Object reference not set to an instance of an object.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.

Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[NullReferenceException: Object reference not set to an instance of an object.]
System.Web.PipelineModuleStepContainer.GetEventCount(RequestNotification notification, Boolean isPostEvent) +30
System.Web.PipelineStepManager.ResumeSteps(Exception error) +266
System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb) +132
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +693

If i continue these requests for another minute or so while the database is still down and then turn the SQL service back on the problem continues. Randomly 1 or 2 requests out of about 50 will load the page but I think there is something being cached with those - I don't think the site actually comes back online for those 1 or 2 requests.

I did implement a work around mentioned in these thread that looks at the unhandled sqlexception in the global.asax and executes a Token.ResetInstance(). I also have it execute this if the Token instance is null or the Token.Instance.User. This work around unfortunately doesn't work. I had high hopes for it because I was thinking the same thing that user was thinking.

Here is the code I used in global.asax:

Code: Select all

        if (Token.Instance == null || Token.Instance.User == null || exception is System.Data.SqlClient.SqlException)
            Token.ResetInstance();

The problem is in the last error i posted (the one that I can't seem to get the site to automatically recover from) is it doesn't even seem to enter any code from the website at all and the call stack seems to support that theory. So there's nothing I can implement to correct the situation at that point. The only thing I can do is manually restart the app pool which does fix the issue but that's obviously not the end solution here is it?

This problem is also mentioned in this article: viewtopic.php?f=42&t=16203&start=0

To me this looks like an issue in AbleCommerce code. Does anyone have a working work-around or fix yet? I'm hesitant to start re-developing AbleCommerce libraries (cause this looks like it will be a significant update in the AbleCommerce libraries/code).

[mtrujillo86]

Anyone ever solve this issue?

Is this a confirmed bug?