Large product file

[palex]

Under Utilities --> Database,
When uploading a product file, the process can be extremely slow if it is a large file. Can this be uploaded / processed via ftp instead? I have between 30k and 50k products.

Thanks!

[Jim]

You can ftp the file to the output directory of your store (where the html files are located) and then select the file on the Utilities > Datatabase > Upload > (pages or products) screen in the top field in the File on server (host): section.

If you have 30 to 50,000 products and your store is on a shared server you are likely to experience extreme slowness. With that many products you should have a dedicated server for just your store.

[palex]

Thanks, Jim.
This works perfectly well, and is actually not slow at all. What might be the cause for the search ceasing to work? I emptied all products, did the upload, published. Now when I do a search (try "Arts"), no results are returned.

Thanks!

[Jim]

I logged in to the store and all of the products that I looked at have the Search Destination set to "Store", but they are not assigned to a store page (there are only 4 under the Pages section). If you want the search results to create a page then you need to set the field to "Made". If they are at some other url then you will need to enter the url and check the radio button to use the url.

There are 2 ways to fix this in your store.
1. use the Power Edit feature to select the appropriate fields (make sure it is just those fields) and if they are all going to have the same value use the Global button and set the appropriate values and save changes. That will change the selected fields in all the selected products to the same value so make sure only those fields are selected or it could mess up your data. (Make a backup before doing this just to be on the safe side)

2. This can be done in an upload of Products by using the fields "Search Dest Type" and "Search Dest" and "Search Make Page". The easiest way to see what the values in the upload file should be is to go to a product and set it up as you want it. Then select that product from the list of products and click the Download button. That will take you to the download screen where you select Tab delimited text file. Then in the Down Load Fields section choose Selected Fields only and click the Select button. From the list on the left select those 3 fields and click the Add button which will move them to the right side. Click ok to close that screen and click the Proceed button at the bottom of the screen to download the file. Save it and then open it in a text editor or spreadsheet program and see what values are there. Then add those columns to your upload file and set the values for each product to the appropriate value as in the file you downloaded. Re-uplaod your products and then publish.

[palex]

Thanks,
That did the trick. Do you know if there is an override for the Display SKU? field? I have everything checked for this field in the product upload file, but everything appears unchecked when viewed for each product.

Also, in the search results, is there a quick way to get rid of the link displayed for each product in the current template?

Thanks again!

[Jim]

Products > Edit Product Layout > is the Display SKU checked? If not then the sku won't be displayed.

I'm not sure what link you are referring to. If it is a more info link then disable the Create moreinfo page on Products > Edit Product Info. If it a link to some page, what setting to you have for the search destination.

[palex]

I posted a screen shot of those links (in black ovals):

http://www-rohan.sdsu.edu/~palexand/ss.png

I am hoping to get rid of these... I'm not sure if there is a quick way to modify the template.

[Jim]

I did a bunch of template testing to see if I could get it to go away and couldn't . I then checked what you had for the search destination and saw that on "Contemporary Art: Art Since 1970, 1/e" it was set to "Made". Since you don't want a page link I changed that to None and did another search. This time that product in the search results does not have the link. Now if you search for "art" the first item should be "Contemporary Art: Art Since 1970, 1/e" and you will see there is no link under it. So change the Search Destination for all your products to "None" and the link won't be displayed.

Your store definitely has problems on the server it is on!!!! I went to the publish screen and clicked the regenerate button and the 4 pages published fine and I saw the messages about indexing products. I noticed that the progress bar stopped at about half way so I looked farther down on the page to see how far it got. It had only indexed about 3,000 products. I went back to the Utilities > Publish screen and there was a message that the previous publish had not completed and there was a Continue button beside the Regenerate button. I clicked Continue and it went for about 15-20 seconds and stopped again. This indicates that the server has a process time limit of somewhere around 20 seconds. I went back and repeated clicking the Continue button probably 15 times and each time it got a little farther but it finally stopped at about 16,000 products. Since it takes longer than 20 seconds to index the products, even with continuing the process, the indexing will never complete. As I mentioned in a previous post or email you will need to get a non-shared hosting account to be able to run a store with that many products. You might talk to your host and see if they will up the timeout limit for your store but my guess is they probably won't.

[palex]

Jim-
Thank you so much for taking time to look at this. Per your earlier suggestion, I had to set the Search Destination to "Made" in order for the items to appear when searched. Now it appears they need to bet set to Store... but I will have another look once the server issue is resolved (the provider is working on this now).

Thanks again!

[Jim]

1. Are you going to have ShopSite generated pages with the products on them?
2. Are you going to have More Information pages for the products?
3. Are you going to have pages not created by ShopSite with the products on them?
4. Do you want ShopSite to automatically create a page for a search result to link to for a product?

The answers to those questions will determine which search destination setting needs to be used and how it is configured.

You would set the Search Destination to the following settings based on the answers to the above questions.
1. Store
2. Moreinfo
3.The specific url of the page
4. Made

[palex]

Hi Jim,
Thanks for all of your helpful info. It looks as if the issue with the server has been resolved, as I now have 40K products that have been generated without freezing / errors.

However, this search issue continues to be problematic. I now know that I need to set all of the Search Destinations to "None" so that a product page will NOT be generated (an answer of NO for each of your four questions). I have done this for all items, but no search results are displayed for any search. I have not changed any other settings. It is most likely something simple I am overlooking. Also, the last time the search was functional, the first several results displayed were not related in any way to the search word (e.g. the first results from searching for "physics" did not contain the word).

Thanks again.

[Jim]

So your answer to each of these questions is "No"?

1. Are you going to have ShopSite generated pages with the products on them?
2. Are you going to have More Information pages for the products?
3. Are you going to have pages not created by ShopSite with the products on them?
4. Do you want ShopSite to automatically create a page for a search result to link to for a product?

If the answer to each of those questions is "No", where is the product information that is being searched for going to be found?

It must be found somewhere or what is the purpose of searching for it? It has to be on SOME web page for a search result to be returned. That page can be a ShopSite generated store page, a product moreinfo page, a page generated by ShopSite just to display the product when it is searched for, or a page that you have created outside of ShopSite with the product info on it. If it isn't on a web page a search from ShopSite will never be able to find it.

[palex]

No, this was working before... there are no product pages. All the information is returned in the search results (title and image).

[Jim]

I went through the code with an engineer to find out what needs to be done as it appears the link is always output for Made pages and doesnot even look at the Link Template Override setting.
Here is what you will have to do.
For all products set the product "Search Destination:" field to the URL radio button but do not enter a URL in the field.
I modified the search link template, JimSearchTest, to not output anything for the link.
I was switching templates on the Preferences > Search Settings > Search Layout screen and don't remember if you had the em_BasicLayout.sst or em_AdvancedLayout.sst set but either will work. The Advanced has lines above and below the price and add to cart which I didn't think looked good so I left it with the em_BasicLayout.sst. but you can changed it to see which you like.

I was testing with the product "125 Physics Projects for the Evil Genius" and using the search term "physics" so you can give that a try and see what you think. If that works for you, then make the change to the Search Destination field for all products by using the Power edit feature. Select the field "Search Make Page" in the left column and the Global option (as long as you don't have any other fields selected) and click Proceed. On the next screen you will see just the Search Destination section where you can click the radio button by the input field at the top and click Save Changes. You will then need to publish the store so the products will be indexed.

Note : The reason the publish started finishing may have been that none of the products were actually being indexed since the search destination was set to "none". I'm not sure what will happen with the publish time with the search destination set to the URL setting.

[palex]

Thanks, Jim.
This does appear to be working, though your suspicion regarding the publishing completion was correct, as it is only making it to about 3,000 (and considerably slower). It is quite strange, because it had been working perfectly with the Search Dest Type = selected, and Search Dest = none.

Thanks again... I will bug the server host again about the publishing issues... this is quite an ordeal!