Sitecore xCommerce Crawlers and Strategies optimization (9.0.3)

Hello,

Last time I had a chance to check search engine in project with xCommerce. Its main responsibility is to collect products and it is based on SOLR services. During checking it, appeared the issue. When the SOLR engine started to indexing products which we had in DB it has eaten whole VM resources and after some time it just hang out. 

Now all of us can say - but why? 

Well it's really the best question we can ask (and the easier one), but unfortunately answer is not so easy to find. To find answer we can start with checking crawling process made by crawlers or strategies

Crawling you say? - Well, yes.

Details:

To be precise: we were working with Sitecore 9.0.2 + SXA 1.8 + xCommerce 9.0.3. We were using commerce API to add products from the external system and in general, it works fine. What we discovered is the difference in indexing implementation between Crawlers which are triggered manually and Strategies which work in the background. As a result, we see differences in SOLR results.

> Strategy that we use is: SellableItemsIntervalAsynchronousStrategy
> Crawler: AllCatalogItemsCrawler.

What is the gap? 

Well, when the strategy is fired up after the product import we see in SOLR many documents of the same product in the same language, like 30, 60 which has the same SitecoreId but different paths with "/[orphan]" as a root. We also figured out that Commerce keeps the same amount of entities in Mappings, where 30 entries have the same SitecoreId but different DeterministicID (also different to its SitecoreId).

We see that Strategy implementation is different than Crawler one and doesn't skip "orphan" items as crawler does. Hence after strategy activity, we see those 30docs in SOLR of 1 product, but after manual rebuilding the index which executes the crawler, we see only 1 (entry which doesn't have an orphan path, but correct one)

Solution:

The easiest solution is to implement skipping orphan entities in Strategies. To obtain that behavior we need to change base class for strategies: "CatalogSystemIntervalAsynchronousStrategyBase"

The code proposal is below.

Thank you !