Magento store owners with the large product catalog generally raise this question:
how many products can Magento actually manage? Today, we will explain the answer by providing practical guidelines.
What are the common mistakes that a person can make when handling a huge Magento catalog?
With and without extra optimizations, how many products different versions on Magento can be easily managed on an average server?
All these details give a rough idea, of course, it refers to a catalog with few categories and simple products. These reports and figures are based on Magento’s own performance and own experiences. The figures may vary greatly after a period of time depending upon the setup and software resources too.
And this brings to notice that, due to Magento’s database design, there are some especially massive aspects that act as an add-on to get the actual number of products Magento really works with.
Some of the most important elements that have a huge impact are:
All this means that a Magento catalog with a few thousand products in a heavyweight catalog setup is concerned in a catalog management.
There are few features that make Magento really flexible, but nothing comes for free.
What makes the difference between Magento versions in terms of catalog performance?
Magento CE 1.x, including 1.9.x and
Magento CE 1.x, including 1.9.x and
Magento 1.x EE
Magento 2 CE
Magento 2 EE
How can Magento be scaled to handle even more products and what are the main bottlenecks for a big catalog?
Indexing
Indexing in Magento is a second step of saving product data and it is the trickiest part when having a huge catalog. It consists of a series of processes to copy product data from database tables optimized for data storage to tables optimized for different aspects of frontend data access. Since indexing is “only” needed for Magento frontend features, it is possible to separate indexing from product save or import. In the newer versions of Magento 1 EE and Magento 2 CE and EE to speed up working in the admin, incremental background indexing is introduced. but still may not be ideal for large volume product updates.
Indexing in Magento 1 CE is one of the greatest bottlenecks.
URL indexing can be bloated to millions of records as tends to provide the most issues, as it runs for a long time and in Magento 1 CE it is not optimized for big catalogs.
Beyond a certain catalog size and number of Magento stores, the increased overhead of product flat indexes will outweigh its benefits so it tends to be better not to turn on this feature for a big catalog.
Catalog search
Many resources need Default MYSQL full-text search for indexing and the actual search on the front end tends to decrease its accuracy and the relevance of the result set is fairly poor. So it is needful to replace, even in Magento 1. There are free alternatives for Magento 1 to replace MYSQL search with Solr, Elasticsearch or Sphinx. Magento 1 EE has Solr and Magento 2 from 2.1 on Elasticsearch. A 3rd party Elasticsearch and Solr 3rd extensions can replace the default MYSQL layered navigation engine at the same time, which can be a huge benefit.
Full Page Cache
Full Page Cache (FPC) is a mechanism whereby HTML pages generated by the server software are cached as a whole.
Next time the same web page is required, the cached version is returned without the need to regenerate the content. No FPC is there in Magento 1 CE and Magento code itself manages the Magento 1 EE caching. It’s too speedy and saves the lot of resources which makes it economical, by implementing caching in a layer in front of Magento is the best way to do FPC and there is no need to touch Magento at all when a cached content is served. This is achieved by Varnish caching in Magento 2. Though Magento 1 CE has no FPC and Varnish, there are good extensions to implement these features, too.
Application Cache
Magento heavily supports different types of configuration and application caches, Redis which is a memory based, scalable application cache is highly recommended with tag management. Magento starting from the latest version of Magento 1 CE creates an extension to handle redis to built.
To sum it up different versions of Magento
What are the common mistakes that a person can make when handling a huge Magento catalog?
There are chances that you can come across additional mistakes while having a huge catalog.
Under scaling :
It is very important to build a live Magento store in such a way that it has plenty of system reserves. Performance tests throughout the development cycles are very useful so that you should know the limit of your system.
Low-quality 3rd party or custom extensions
Generally, 3rd party extensions are not meant for testing the large catalogs. Sometimes even a small oversight in the extension design can harm the performance.
Lack of proper monitoring
You need to monitor your system at a regular interval to avoid issues and bottlenecks.
Some technical insights
Data Storage in Magento
Magento is very complex and its features are virtually maxing out the limits of a PHP/MYSQL based system. Products are urged along with it comes the number of attribute scheme through prices, images, categories, product options to different product relations.
All these properties have different layers of values as many stores and languages used. Moreover, these aspects are potentially extended by a number of 3rd party extensions. Thus, it concludes that saving and updating products in the database is complex action.
Magento’s Frontend Data Access – Indexes
In Magento, indexing is a process where data is copied from the database tables optimized for data storage to tables optimized for frontend data access.
The structure of the database tables where product data is primarily saved is optimized for flexibility for data storage but as nature is complex in rational databases which are not optimized for different types of data retrieval at the same time. To reduce all this Magento has introduced Index tables which are known by the name of indexing.
You need some indexes for your stock Magento to work like URL, category/product relation, price and stock and attribute indexes and also search index. Apart from these, there are some optional indexes such as catalog flat and product indexes which flatten the EAV and multi-store data to dedicated store tables and single product rows.
Indexing in Magento comes with two faces first; it enables Magento’s most powerful features to work and optimizes data access and second, it makes it more complex, time-consuming and resource greedy to store product data in a usable way for the frontend.
Product Attributes Index :
It is just as layered navigation will work as it copies product/attribute options data to a table structure that is optimal for finding products based on the different attribute options which are already included. The number of attribute options is multiplied by the number of stores/languages Magento has.
Product Price Index :
By default, it is important so that sorting and filtering product prices work. Tier prices, configurable, bundle, and product complicate this calculation since the minimal price of complex product types depends on the price of the constituent products. The number of Magento websites and price groups complicates these factors.
Catalog URL Rewrites Index :
This index is essential so that SEO friendly URLs and redirection from old URLs to new ones work. The table is greatly affected by the depth and size of the category structure, the number of old URLs and stores/languages. The size of the table can go up to millions of records and later also it will keep on growing even in the case of a catalog with a few thousand products and few languages. As far as big catalogs are concerned Magento 1 CE and Magento 1 EE up to 12.x are facing the most problematic indexer.
Category Products Index :
Based on categories you have to optimize product filtering by storing catalog-product relations in a separate table. It is essential for the frontend.
Catalog Search Index :
For MYSQL based search you need this index. It merges option labels of individual products and the text of product attributes so that you can easily search in MYSQL full-text engine. Due to many stores, their many index rows are created for the single product.
Stock Status Index :
This index is used for calculating the fact whether the product is purchasable in Magento which can be governed by the mixture of some global, website and product level values and settings.
Product Flat Data Index :
This index is optional and was part of Magento in later stages of product listing/sorting and spare server resources. Its job is copied product attribute values which can be normally retrieved by huge queries of joining multiple tables into a flat structure with only one record per product and store.
Database Design
Magento EAV
EAV stands for Entity-Attribute-Value. Is a dynamic attribute management pattern which enables adding/removing/modifying product attributes like color, manufacturer etc., without changing the structure of the database tables. It is a very powerful and user-friendly feature and Magento has a number of configurational options for custom attributes out of the box.
Magento Multi-Store :
It is a unique Magento feature that a single Magento database can manage multiple stores in a way that attributes can have dedicated store level values that may override default values. This, again, is a highly user-friendly approach and makes it easy, for example, to create different language versions of the same catalog only by changing the description of the products and label of the product options.
Magento Indexing :
In Magento, Indexing is a process where data is copied from database tables and optimized for data storage to tables and then optimized for frontend data access. Different indexes are used for different types of data access. Indexing processes are built to speed up the shop and generally provide a huge benefit, in the case of full reindexing, which is sometimes inevitable, and any Magento shop should be prepared to be able to perform a full index rebuild, it may take a long time and require huge resources to reindex all product data.
Bottom Line
Magento has seen a great evolution since the beginning and now with Magento 2 which is not only the most feature rich open source online store system but also it is considered best in performance. If proper care and resourced are provided then huge catalogs can be managed easily and your business might go on increasing.
You can raise revenue of your business by setting up...
Now it is much easier to configure Magento 2 translation....
Do you know customer Chat Facebook is a free application...
Let our Magento expert connect to discuss your requirement.
We offer Magento
certified developers.
Our Magento clientele
is 500+.
We sign NDA for the
security of your projects.
We’ve performed 100+
Magento migration projects.
Free quotation
on your project.
Three months warranty on
code developed by us.