Scaling spaces

This topic discusses the recommended limits of workspaces in a space and the implications of going beyond those limits.

Performance lab results

Tests show that for best performance, the number of workspaces in a space should not exceed 200.

Creating more than 300 workspaces in a space can result in significant performance degradation.

The following sections describe implications of going beyond the recommended limits.

Following are system architecture aspects of the recommended limits:

ValueEdge's internal architecture is optimized to support a large number of spaces, where the spaces have less than 200 workspaces. This accommodates the main usage pattern of the large majority of customers.

The system uses in-memory cache per space. More workspaces in a single space translates to higher memory consumption. There is also less chance of optimization when individual workspaces are no longer in use.

RDBMS (such as MSSQL) caches are managed at the DB schema level. The DB schema in ValueEdge represents a space. When there are a large number of workspaces in a single schema, the DB cache has less chance to optimize performance of DB operations.

DB indexes are much bigger in the scope of a single large space. Overall DB performance may be affected. The number of connections opened per schema will significantly increase and can be a bottleneck for concurrent requests. This can affect the overall performance and end user experience.

The same is true of Elasticsearch whose indexes are also managed at the space level. Elasticsearch performance in ValueEdge is optimized for multiple spaces, rather than one very large space. The implications of having a very large space can include the following:
- A high likelihood of performance problems.
- A longer reindex when Elasticsearch version upgrade is required.

Smaller spaces allow for horizontal scaling of ValueEdge. Large spaces force vertical scaling at higher risk and higher cost of maintenance over time. The following system components do not allow horizontal scaling:
- RDBMS (such as MSSQL): It is possible to split different spaces across different DBs. Splitting a single space's workspaces between different DBs is not supported.
- File system. You cannot split the file system storage of workspaces in the same space.

Async task execution architecture, and specifically its fairness strategy, is performed mainly at the space level. This means that async tasks running in parallel in a single large space will take longer to execute when compared to many smaller spaces.

Following are upgrade aspects of the recommended limits:

Upgrade in ValueEdge is designed to run in parallel on several spaces at the same time. Fewer workspaces per space will allow faster parallel upgrade of spaces. Larger spaces will result in longer upgrade times.
Staging and validations of a large space before a production upgrade will be more expensive, as it will involve restoring and validating large spaces, instead of being able to restore and validate one smaller space at a time.
In case an upgrade fails, all users in the same space will be impacted. The more spaces, the smaller the number of impacted users.
Because space becomes available only when it was fully upgraded, no users will be able to access the system until the full upgrade is complete. With many smaller spaces, once a space upgrade has finished, all users of that space can start working, even if there are still upgrades running in the background on other spaces.

Following are run-time aspects of the recommended limits:

Certain user actions, such as large bulk-updates, can impact the entire space, essentially blocking all operations on a specific DB table until the action has completed.
- The more workspaces you have in a single space, the greater the chance of such an event happening. The impact will be on the entire user base in the same space.
Overall, read operations will experience reduced performance the larger the space schema is. This results from the RDBMS having to go process larger data sets.
All workspaces within a space share some common assets. These include: Roles, API access, space parameters, pipeline purge policy, and login notifications. It can become impractical to apply the same shared settings across a multitude of workspaces, where some of the workspaces are unrelated to others.

Following are backup and recovery aspects of the recommended limits:

Backup and restore are performed at a space level. The bigger a space is, the longer backup and restore operations will take.
If there is a need to restore a single workspace, due to data deletion or corruption, a full space restore is required before a single workspace can be extracted from it. Additionally, such a restore reverts the data for all workspaces in the space.
With larger spaces there will always be some data that is still required and in use. This makes it virtually impossible to deprecate spaces that are no longer in use.