IIS errors related to CustomURLs

[crockettdunn]

Good day,

I restarted my website a couple times today to make an error stop happening. I noticed the site was stable for about an hour, then the errors resumed.

In the server Event Viewer, I see loads of errors like the following during the times the site is "broken" (broken, here, means returning many 500 errors for pages that will work again after I restart the web site in IIS.

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 9/23/2013 11:40:04 PM
Event time (UTC): 9/24/2013 3:40:04 AM
Event ID: ca233a9ac27848dc8bc1473f8b6ef1ef
Event sequence: 40933
Event occurrence: 14982
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT-1-130244420185527021
Trust level: Full
Application Virtual Path: /
Application Path: C:\Users\store.hgmforkliftparts.com\wwwroot\
Machine name: CDWS2

Process information:
Process ID: 3260
Process name: w3wp.exe
Account name: CDWS2\AppPoolUser

Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at CommerceBuilder.Catalog.DefaultUrlRewriter.a.a(CustomUrl A_0)
at System.Collections.ObjectModel.KeyedCollection`2.RemoveItem(Int32 index)
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.c.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)



Request information:
Request URL: http://store.hgmforkliftparts.com/mitsu ... 02602.aspx
Request path: /mitsubishi_forklift_parts/mb32b45-02602.aspx
User host address: 66.249.75.48
User:
Is authenticated: False
Authentication Type:
Thread account name: CDWS2\AppPoolUser

Thread information:
Thread ID: 6
Thread account name: CDWS2\AppPoolUser
Is impersonating: False
Stack trace: at CommerceBuilder.Catalog.DefaultUrlRewriter.a.a(CustomUrl A_0)
at System.Collections.ObjectModel.KeyedCollection`2.RemoveItem(Int32 index)
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.c.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Custom event details:


Can anyone help me with this? I don't understand what this error message is telling me.

Thanks,
Crockett

[crockettdunn]

Well, I can definitely replicate the failure when I spider the site with Xenu, but I don’t have any intelligent way of auto-restarting IIS or repairing the underlying problem.

[jmestep]

It might be that the only thing you can do is block Zenu via robots.txt. I had to do that to some spiders in Able 5 because they were killing the site.

[crockettdunn]

thanks jmestep- but I didn't mean to imply I had diagnosed the problem as being caused by Xenu. *replicated* with xenu, but it may be a velocity issue, or a url pattern issue. I hasn't considered a "rogue" user. I usually leave that stuff to my web host, but it's something else to consider... ESPECIALLY if it can be fixed with a velocity filter. hmmm....

[crockettdunn]

This appears to be a load issue. I set the app pool to recycle at 400,000K physical memory. It's recycling every 2-4 hours with this setting, which isn't ideal.

I tried to get load balancing working via web garden with 2 worker processes, but I began getting "validation of viewstate mac failed" messages when trying to get the session state working out of process.

Anyone out there running load balancing via multiple worker processes?
This resolved the issue when load testing using Xenu, but "real" users need to maintain session state, and I was unable to make that happen out of process.

[jmestep]

If you put the same machinekey settings in both web.config files, you shouldn't get that error. Without it in the web.config, it gets picked up from the servers' machinekey. Just be sure to back up your encryption keys and be ready to restore them before you change the keys.

[crockettdunn]

status
opened a support ticket for this- I've been trying to replicate the errors with load, but for all I know it could just be dumb luck that the site breaks whilst I'm also loading it. Anyway I'm limping along recycling the app pool whenever pages start mostly returning the custom URL 500 error. Right now I'm running the SQL tuning advisor with a recent trace file. I have no experience with DB tuning, so I'll see what Mike has to say about its recommendations.

I may continue with the pseudo load balance with two worker processes if I can fix the machine key issue per jmestep's recommendations (thanks).

desperate and temporarily stumped...

[crockettdunn]

on a dev server, i got out-of-process state working, with the hope that at the very least I can auto-recycle the app pool without kicking users' sessions. But, the load time seems to be 4x as long.

[crockettdunn]

Been a couple weeks, and I used some AC paid support. I currently have state running out-of-process, and 4 worker processes for the app pool (theoretically balancing the load?).

I used the MS SQL Tuning Advisor, and 80-90% of the problems went away.

Now I'm left with 3 types of errors in the Application log, and errors "maintaining custom URLs" in the AC log.

Here are the remaining 3 errors which repeatedly occur:

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 10/11/2013 6:50:04 PM
Event time (UTC): 10/11/2013 10:50:04 PM
Event ID: 5d7464f4359b48909b3cf2c68a64d212
Event sequence: 1121
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT-1-130260031223167550
Trust level: Full
Application Virtual Path: /
Application Path: C:\Users\store.hgmforkliftparts.com\wwwroot\
Machine name: CDWS2

Process information:
Process ID: 4224
Process name: w3wp.exe
Account name: CDWS2\AppPoolUser

Exception information:
Exception type: ArgumentException
Exception message: An item with the same key has already been added.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.ObjectModel.KeyedCollection`2.InsertItem(Int32 index, TItem item)
at CommerceBuilder.Catalog.DefaultUrlRewriter.InitializeCache()
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)



Request information:
Request URL: http://store.hgmforkliftparts.com/Produ ... Id=1105536
Request path: /Product.aspx
User host address: 66.249.75.48
User:
Is authenticated: False
Authentication Type:
Thread account name: CDWS2\AppPoolUser

Thread information:
Thread ID: 39
Thread account name: CDWS2\AppPoolUser
Is impersonating: False
Stack trace: at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.ObjectModel.KeyedCollection`2.InsertItem(Int32 index, TItem item)
at CommerceBuilder.Catalog.DefaultUrlRewriter.InitializeCache()
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Custom event details:




Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 10/11/2013 6:12:04 PM
Event time (UTC): 10/11/2013 10:12:04 PM
Event ID: 9d6c16dffc2e4036adc56daf7a64bd2f
Event sequence: 3
Event occurrence: 1
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT-1-130260031230811599
Trust level: Full
Application Virtual Path: /
Application Path: C:\Users\store.hgmforkliftparts.com\wwwroot\
Machine name: CDWS2

Process information:
Process ID: 4640
Process name: w3wp.exe
Account name: CDWS2\AppPoolUser

Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.ObjectModel.KeyedCollection`2.InsertItem(Int32 index, TItem item)
at CommerceBuilder.Catalog.DefaultUrlRewriter.InitializeCache()
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)



Request information:
Request URL: http://store.hgmforkliftparts.com/yale_ ... 0252c.aspx
Request path: /yale_forklift_parts/yt580000252c.aspx
User host address: 66.249.75.48
User:
Is authenticated: False
Authentication Type:
Thread account name: CDWS2\AppPoolUser

Thread information:
Thread ID: 7
Thread account name: CDWS2\AppPoolUser
Is impersonating: False
Stack trace: at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.ObjectModel.KeyedCollection`2.InsertItem(Int32 index, TItem item)
at CommerceBuilder.Catalog.DefaultUrlRewriter.InitializeCache()
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Custom event details:



Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 10/17/2013 2:41:54 PM
Event time (UTC): 10/17/2013 6:41:54 PM
Event ID: 7d9ce4f5a012448f9e7c1f7fc86d66ed
Event sequence: 6937
Event occurrence: 3
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/2/ROOT-1-130264925423239998
Trust level: Full
Application Virtual Path: /
Application Path: C:\Users\store.hgmforkliftparts.com\wwwroot\
Machine name: CDWS2

Process information:
Process ID: 3312
Process name: w3wp.exe
Account name: CDWS2\AppPoolUser

Exception information:
Exception type: ArgumentOutOfRangeException
Exception message: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
at System.Collections.ObjectModel.Collection`1.RemoveAt(Int32 index)
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)



Request information:
Request URL: http://store.hgmforkliftparts.com/mitsu ... 26108.aspx
Request path: /mitsubishi_forklift_parts/mb26108.aspx
User host address: 66.249.75.48
User:
Is authenticated: False
Authentication Type:
Thread account name: CDWS2\AppPoolUser

Thread information:
Thread ID: 12
Thread account name: CDWS2\AppPoolUser
Is impersonating: False
Stack trace: at System.Collections.ObjectModel.Collection`1.RemoveAt(Int32 index)
at CommerceBuilder.Catalog.DefaultUrlRewriter.a(String A_0)
at CommerceBuilder.Catalog.DefaultUrlRewriter.RewriteUrl(String sourceUrl)
at CommerceBuilder.Services.b.c(Object A_0, EventArgs A_1)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


Custom event details:


Any experts out there know what's going on? My hunch is that two things are trying to modify the same collection simultaneously.

[ForumsAdmin]

At least one of the exceptions indicates that the synchronization of multiple threads may be failing at some point. Which version of AbleCommerce are you at?

[crockettdunn]

7.0.7 build 14600

[Katie]

Have you applied any hot patches after build 14600 was released? One of the errors (An item with the same key has already been added) shows up in this list:

http://help.ablecommerce.com/index.htm# ... atches.htm#

I'd recommend apply all the patches on this page, as long as they apply to you.

[crockettdunn]

Yes I've applied all of those patches. The context of my "item with same key" error appears to be the URL rewriter. The patch deals with this error for inventory management, which we are not using.