Large catalog timeout generating Sitemap file
Large catalog timeout generating Sitemap file
I've got a client with 94,000+ products in the catalog, each one has a digital good.
When we run the sitemap generator page, it times out every time. I've increased the timeout from 2 minutes all the way up to 20 minutes. Still, the page times out.
Any suggestions on improving the routine so it will run in a reasonable time? The catalog is large, but it shouldn't take that much time to build a sitemap file.
When we run the sitemap generator page, it times out every time. I've increased the timeout from 2 minutes all the way up to 20 minutes. Still, the page times out.
Any suggestions on improving the routine so it will run in a reasonable time? The catalog is large, but it shouldn't take that much time to build a sitemap file.
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
Re: Large catalog timeout generating Sitemap file
What happens when feed is generated automatically without manually triggering it from page? In feed there is some code related to price calculation and variants, I wonder if you try commenting out that part obviously just for testing and see if it makes any difference. Is there any error in log ?
Re: Large catalog timeout generating Sitemap file
That was my question too. There is no file being auto-generated. And there's no error in the error log, which is confusing because there should be something there. Nothing in app.log either.mazhar wrote:What happens when feed is generated automatically without manually triggering it from page?
The error is just 'Request timed out.'. It seems like it's running, it's just taking an enormous amount of time to do it.
Is the sitemap generator code in /app_code/ ? I don't have full source for this client.
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
Re: Large catalog timeout generating Sitemap file
Joe... did you ever figure this out? I have a site doing the same thing all of a sudden. Also has a very large catalog.
Re: Large catalog timeout generating Sitemap file
After much investigation... I figured out it was simply too many products in the DB (around 250,000)... so I switched out this code:
With this code:
in App_Code\CommonSiteMapProvider.cs
It wasn't a database timeout or bad product or circular reference etc. I think the page was timing out waiting for the code behind to run.
Code: Select all
foreach (Product prod in store.Products)
{
if (prod.Visibility == CatalogVisibility.Public)
{
url = UrlGenerator.GetBrowseUrl(prod.Id, CatalogNodeType.Product, prod.Name);
siteMapItems.Add(new SiteMapUrl(GetAbsoluteUrl(url), options.DefaultChangeFrequency, options.DefaultUrlPriority));
}
}
Code: Select all
ICriteria criteria = NHibernateHelper.CreateCriteria<Product>();
criteria.Add(Restrictions.Eq("VisibilityId", (byte)CatalogVisibility.Public));
IList<Product> _Products;
_Products = ProductDataSource.LoadForCriteria(criteria);
foreach (Product prod in _Products)
{
url = UrlGenerator.GetBrowseUrl(prod.Id, CatalogNodeType.Product, prod.Name);
siteMapItems.Add(new SiteMapUrl(GetAbsoluteUrl(url), options.DefaultChangeFrequency, options.DefaultUrlPriority));
}
It wasn't a database timeout or bad product or circular reference etc. I think the page was timing out waiting for the code behind to run.
Re: Large catalog timeout generating Sitemap file
Interesting. Did you have a lot of products where visibility was not set to public?
I almost wonder if the difference was in how you're calling LoadForCriteria() directly as opposed to how nHibernate populates the child collection of store.Products. I wouldn't expect them to be any different in terms of performance, but perhaps it is.
250,000 is quite the catalog size
I almost wonder if the difference was in how you're calling LoadForCriteria() directly as opposed to how nHibernate populates the child collection of store.Products. I wouldn't expect them to be any different in terms of performance, but perhaps it is.
250,000 is quite the catalog size
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
Re: Large catalog timeout generating Sitemap file
Yes, he has many products that are not public. Product samples, fabric remnants, etc. I probably need to get with him and remove some if possible but he might want to save the order history on them. Would have to look into that.
I was wrong on the product count. ProductId is up to 238K but only 168K actual records. Only 42,000 are public, 84,000 are hidden and 42,000 are private.
So pulling it the way I am now I cut it down to about 25% of the total.
I was wrong on the product count. ProductId is up to 238K but only 168K actual records. Only 42,000 are public, 84,000 are hidden and 42,000 are private.
So pulling it the way I am now I cut it down to about 25% of the total.
Re: Large catalog timeout generating Sitemap file
That's what I figured.
You might gain a little boost to query speed if you added an index on the VisibilityId field in ac_Products. Might not help much, but easy to at least try and see.
You might gain a little boost to query speed if you added an index on the VisibilityId field in ac_Products. Might not help much, but easy to at least try and see.
Joe Payne
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com
AbleCommerce Custom Programming and Modules http://www.AbleMods.com/
AbleCommerce Hosting http://www.AbleModsHosting.com/
Precise Fishing and Hunting Time Tables http://www.Solunar.com