Version 2022.2. List of optimisations
- Switch to Apache Kafka in asynchronous processing of transactions and events in the payment system
- Optimization of batch operations
- A new way to store purchases: purchase information in the form of a JSON document
- Improving the performance of offers
- Updating indexes
- Optimization of methods for obtaining client attributes
- Restrictions on the total volume of customers have been lifted
- Regular cleaning of purchase data older than the specified date
- Optimization of purchase information
Switch to Apache Kafka in asynchronous processing of transactions and events in the payment system
To improve the performance of the payment system, the logic of working with queues in it has been changed.
- It was: all new transactions were sent directly to the database, after which they were processed in the background by a special background service.
- It became: all new transactions end up in a special distributed streaming platform Kafka.
The platform Kafka allows you to:
- increase the fault tolerance of the System by distributing data to different nodes;
- horizontally scale the load;
- due to the high throughput, absolutely all events passing through it can be processed in real time;
- thus, do not accumulate queues for transferring funds between accounts.
Optimization of batch operations
It was possible to significantly increase the speed of batch operations due to:
- changes in the logic of the batch operations themselves;
- deleting tables from the database that record data on batch operations:
- only errors are now presented in the records,
- Unused indexes have been removed.
The table below shows the results obtained during the comparison of batch operations before and after optimization.
| Batch operation | Number of processed data | Before optimization | After optimization |
|---|---|---|---|
Importing card numbers | 300,000 card numbers | 6 hours | 3 to 11 minutes |
| Import of customer attributes required for registration | 300,000 customers | 30 minutes | 10 minutes |
| Import of optional customer attributes | 300,000 customers | 6 hours | 11 minutes |
| Import passwords | 300,000 passwords | 5 hours | 11 minutes |
| Customer registration | 300,000 customers | 9 hours | 10 minutes |
The speed of batch reading of messages from Rabbit queues has also been optimized.
A new way to store purchases: purchase information in the form of a JSON document
The way of storing purchases has changed. Now, instead of a relational purchase storage model, information about positions, operations and processing results is stored in the form of a JSON document. This made it possible to speed up the insertion of this information and overcome the limit of 200 receipts per second (up to 500 checks).
Improving the performance of offers
To increase the speed of the offers, the following optimizations were carried out:
- The campaign parameters (filters and actions) have been completely switched to caching, which allowed faster access to them;
- Internal blocking of offers from the status system has been eliminated, which has allowed to increase the speed of offers with status systems.
Updating indexes
- 15 unused indexes have been deleted. This reduced the overall size of the tables, and also allowed a slight increase in the insertion speed.
- 6 indexes were optimized, which made it possible to increase the performance of some queries by reducing the load on the database core.
Optimization of methods for obtaining client attributes
To increase the speed of processing requests to obtain information about customer attributes:
- Optimized tables in the database (by reducing the size of the table by 2 times);
- Improved filtering, which now occurs strictly by indexes.
Restrictions on the total volume of customers have been lifted
The limit on the total volume of customers of 200 million has been removed from the System.
Regular cleaning of purchase data older than the specified date
To optimize the storage of information in Databases (DB) and for timely cleaning of information, a regular task "Event Cleaning" was done.
When running a regular task, the process of deleting information from the business activity table in the database occurs. This way, the database is automatically cleared of old records (events that have already been completed in the System).
The results obtained during the regular work are presented in the table:
| Description | |
|---|---|
| Duration of work | During one cycle of a regular task with a duration of one hour, information about 1,500,000 events was deleted from the Database. The approximate amount of deleted information is 2.5 GB. |
| Results of background work of a regular task |
After the end of the regular task, the memory was restored to its original value. |
| Impact on the work of your Personal Account and mobile application | A regular task does not affect the work of your Personal Account and Mobile Application. |
| Impact on BI data | The data in BI remains unchanged. |
| Impact on the Marketing management platform (MMP) |
|
Optimization of purchase information
The database table related to purchases has been optimized:
- Information about purchases (for example, the receipt number, purchase ID, check items, etc.) is stored in a compressed form.
- The name of the offer is not saved in the table, only its identifier.
This allows you to reduce the amount of data storage.