Token.Instance.User is suddenly null, crashed site

For general questions and discussions specific to the AbleCommerce 7.0 Asp.Net product.
Post Reply
LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Wed Mar 14, 2012 11:52 am

In the middle of the night, for no apparent reason that I can see, sql login failed for the login I'm using on my site, and then it came back up and started working again, but then Token.Instance.User in AdminPanel.ascx.cs was suddenly null, throwing errors and the entire site went down because of it. It seemed to happen immediately after a database backup, but I'm doing a backup every night at that time, and never had an issue before. I restarted IIS and the problem was fixed. I don't want this to happen again, as my site was down for several hours overnight before I woke up and saw the alert in my email that came in at 1am.

Does anyone have any ideas why this would happen?

Thanks for your help.

User avatar
david-ebt
Captain (CAPT)
Captain (CAPT)
Posts: 253
Joined: Fri Dec 31, 2010 10:12 am

Re: Token.Instance.User is suddenly null, crashed site

Post by david-ebt » Wed Mar 14, 2012 2:40 pm

We've seen a similar problem with the Token.Instance.User being null after a database problem. It most frequently happens in our development environment because someone may be doing other work on the SQL Server which makes it unavailable. The only fix is to wait for the application to timeout on IIS or reset IIS as you did.

Can't tell you for sure why this happens, but I think it is related to the Store settings being cached on the web server.
David
http://www.ecombuildertoday.com
Enhanced Reporting for AbleCommerce
Image

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Wed Mar 14, 2012 2:49 pm

Thanks David, very helpful. I've noticed it to from time to time in my dev environment, but never really paid it much mind as things get messed up when you're doing dev and strange things always happen.

If the store settings are cached on the webserver, is there a reason why they are cached? How do I change that? I assume this will affect performance?

User avatar
david-ebt
Captain (CAPT)
Captain (CAPT)
Posts: 253
Joined: Fri Dec 31, 2010 10:12 am

Re: Token.Instance.User is suddenly null, crashed site

Post by david-ebt » Wed Mar 14, 2012 3:13 pm

The store settings (and some other stuff) are cached so that the web site doesn't have to do database calls for every page and even a lot of user controls to get things like the store name, URL, SSL. localization variables, etc. This significantly cuts down on database calls and since the data is pretty static, it makes great sense to cache. To remove the cache you would need the Able source code and removing the cache would have a significant impact on performance.

It would be possible to create a service or console application that ran on your web server that would monitor the web site and perform an IIS reset or application pool restart. You can take a look at this post for ideas (http://stackoverflow.com/questions/8197 ... mmatically) on how to do that.

If you don't have an automated monitor on your site, there are services that will tell you whether a site is working or not. You could have them alert you so at least you can go manually reset IIS if there is a problem. This might be the simplest approach since it doesn't sound like this has happened before. You'll need to compare how much business you're losing to the cost of monitoring or developing a reset application.
David
http://www.ecombuildertoday.com
Enhanced Reporting for AbleCommerce
Image

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Wed Mar 14, 2012 3:17 pm

Developing a reset application sounds like the way to go. I do have monitoring on the site, that what was how I knew, it just happened when I was asleep, so didn't know it until I woke up this morning and saw the email. I've scheduled an application pool reset for every night right after the backup, I had it scheduled for earlier, so hoping that will help, but the reset app I think will be good. Thanks for the link, I'll look into it.

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Tue Mar 20, 2012 5:22 pm

So I've done quite a bit of research into this issue as it happened again last night. What I've come to the conclusion is that the application pool recycle is what is killing the site, not the backup. And it's not consistent. Sometimes it crashes and sometimes it's fine. Does anyone have any idea why this would happen?

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Thu Mar 22, 2012 11:33 am

After even more research we've found the issue. The server had automatic windows updates on. So it was rebooting in the middle of the night at exactly the time the website would go down. Because the sql server was unavailable, the site would crash and only an IIS restart would fix the issue. We originally had our sql database on our webserver and just recently (about a month ago) moved to a two server (sql and web) solution and we neglected to turn off the automatic updates on the sql server like we had the webserver when we first built it. It's only been a day since we found this issue, so if we experience more crashes like this I'll repost.

Thanks everyone for all your help. I do feel like this is a bug in AbleCommerce. If the Sql Server is not available, the site should not blow up like this when it comes back online. I did notice that this issue was supposed to have been fixed in 7.0.5, and we are running 7.0.7, maybe it was a little bit different of an issue, unsure, but still feel this should not happen.

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Mon Mar 26, 2012 1:38 pm

So, I'm ready to pull my hair out. The site crashed again last night, so it was not the server reboot from updates that was causing it. The same error is in the log about the login failed for explicitly specified database, then Token.Instance.User is null and the site is down until IIS is restarted.

It crashed last night at exactly the same time the application pool was restarted. Originally it was doing this, but the database backups and the windows updates were all happening at the same time, so I couldn't tell which it was. I changed the windows updates to not update unless I do it manually, and changed the application pool to recycle at 5am instead of 3am when the backups happen.

So, now I'm back to thinking that the app pool recycle is causing the issue, but I have no clue why this would happen. I see the app pool recycle in the log at 5:00:02AM, and the first website error comes at 5:00:14AM.

Any ideas? The only thing I could find was that I had auto_close set to true on my database, which I changed today, but I have no idea if that will fix the issue or not.

sloDavid
Lieutenant Commander (LCDR)
Lieutenant Commander (LCDR)
Posts: 92
Joined: Thu Feb 25, 2010 12:34 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by sloDavid » Wed Apr 11, 2012 12:53 pm

Now that a couple weeks have gone by, did that fix the issue for you?

LauraNichols
Lieutenant, Jr. Grade (LT JG)
Lieutenant, Jr. Grade (LT JG)
Posts: 28
Joined: Tue Sep 27, 2011 2:11 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by LauraNichols » Wed Apr 11, 2012 12:56 pm

Yes, been 2 weeks with no issues. I think it was a combination of the auto_close being set to true and the windows updates restarting the server. It's still a very bad bug though. The application should be able to recover if it cannot connect to the database once the database comes back up. I mean come on, standard best practices here.

crazyjoe
Commander (CMDR)
Commander (CMDR)
Posts: 172
Joined: Mon Apr 26, 2010 2:20 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by crazyjoe » Tue Sep 18, 2012 8:18 am

Hi Laura, I'm having this issue hit me constantly now. Can you tell me how you set auto_close to false in your DB. I've looked around the web and found a bunch of ways ridiculously complicated ways to do it but I am hoping you could point me in the direction that you used. Thanks!
Crazy Joe Sadloski
Webmaster
Hot Leathers Inc.
http://www.hotleathers.com

User avatar
jmestep
AbleCommerce Angel
Posts: 8164
Joined: Sun Feb 29, 2004 8:04 pm
Location: Dayton, OH
Contact:

Re: Token.Instance.User is suddenly null, crashed site

Post by jmestep » Tue Sep 18, 2012 1:41 pm

If you right click the database in SQL server to get to properties, autoclose setting is under the options tab.
Judy Estep
Web Developer
jestep@web2market.com
http://www.web2market.com
708-653-3100 x209
New search report plugin for business intelligence:
http://www.web2market.com/Search-Report ... -P154.aspx

crazyjoe
Commander (CMDR)
Commander (CMDR)
Posts: 172
Joined: Mon Apr 26, 2010 2:20 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by crazyjoe » Tue Sep 18, 2012 3:22 pm

Do you know of any PCI Compliance issues or security issues this may open us up to? My hosts were weary about doing this for me because (as they said) "AutoClose property exposes server behavior for databases not accessed by a user. If TRUE, the database is closed, and its resources are freed when no user connection accesses the database. If FALSE, the server maintains the database in an open and ready state regardless of user activity." That sounds like a bunch of database mumbo jumbo to me, can you tell me if it's anything I should worry about? I just want to ensure I get my this crazy problem fixed.

Thanks Judy! I always appreciate your help!
Crazy Joe Sadloski
Webmaster
Hot Leathers Inc.
http://www.hotleathers.com

User avatar
stewie1570
Ensign (ENS)
Ensign (ENS)
Posts: 3
Joined: Thu Aug 30, 2012 3:49 pm
Location: Salt Lake City, UT
Contact:

Re: Token.Instance.User is suddenly null, crashed site

Post by stewie1570 » Tue Sep 25, 2012 2:01 pm

Setting the AutoClose to true or false on my localhost environment has no affect on this AbleCommerce bug. I can still re-create the problem and it doesn't recover when the database comes back online with either value for AutoClose on the db.

crazyjoe
Commander (CMDR)
Commander (CMDR)
Posts: 172
Joined: Mon Apr 26, 2010 2:20 pm

Re: Token.Instance.User is suddenly null, crashed site

Post by crazyjoe » Tue Sep 25, 2012 2:16 pm

Hi Stewie, I have been error free for a full week now. Here is what my hosts said they did, maybe it will help you...
There were a couple tables that were owned to the schema of your user and not dbo. I reowned all objects to DBO and modified the SQL user to have use the DBO schema instead of the schema based on the username. I also increased the amount the database grows.
Good luck!

( this was also posted in this thread... viewtopic.php?f=42&t=16233 )
Crazy Joe Sadloski
Webmaster
Hot Leathers Inc.
http://www.hotleathers.com

User avatar
stewie1570
Ensign (ENS)
Ensign (ENS)
Posts: 3
Joined: Thu Aug 30, 2012 3:49 pm
Location: Salt Lake City, UT
Contact:

Re: Token.Instance.User is suddenly null, crashed site

Post by stewie1570 » Tue Sep 25, 2012 2:37 pm

Thanks Joe,
I'll give that a try. I'm thinking that's not my issue though because if it were permissions I would think that problem would persist after an application pool restart which my problem goes away after an application pool restart. So for me this exact issue starts when the db goes down (for whatever reason) but when the db comes back up the site is still down and is still reporting this error until the app pool restart.

User avatar
jmestep
AbleCommerce Angel
Posts: 8164
Joined: Sun Feb 29, 2004 8:04 pm
Location: Dayton, OH
Contact:

Re: Token.Instance.User is suddenly null, crashed site

Post by jmestep » Wed Sep 26, 2012 7:05 am

So for me this exact issue starts when the db goes down (for whatever reason) but when the db comes back up the site is still down and is still reporting this error until the app pool restart.
This basically what I am seeing on local test sites. For some reason my SQL Server doesn't start automatically anymore after a computer restart so I get the above symptoms.
Judy Estep
Web Developer
jestep@web2market.com
http://www.web2market.com
708-653-3100 x209
New search report plugin for business intelligence:
http://www.web2market.com/Search-Report ... -P154.aspx

User avatar
Katie
AbleCommerce Admin
AbleCommerce Admin
Posts: 2651
Joined: Tue Dec 02, 2003 1:54 am
Contact:

Re: Token.Instance.User is suddenly null, crashed site

Post by Katie » Tue Oct 09, 2012 6:42 am

Good Morning,

A fix has been provided for this issue. Please go to this page in our help site for more information -

http://help.ablecommerce.com/index.htm# ... atches.htm

Thanks for everyone's efforts in troubleshooting this issue.

Katie
Thank you for choosing AbleCommerce!

http://help.ablecommerce.com - product support
http://wiki.ablecommerce.com - developer support

Post Reply