Better utilize database indexes to improve performance #910
Description
Our database implementations for both SQL and Dexie.js are only as optimized as they need to be for current usage/demand of Mesh. However, there are a few known areas where would could make further optimizations if needed in the future.
For SQL, we do not currently use any indexes (except for primary keys). Indexes can improve query performance by reducing the number of disk reads required. However, indexes also have an up front cost since they make every write slightly slower. Care should be taken to ensure that we only create indexes as needed, and that the indexes we create actually do improve performance in a meaningful way. We should use the slow query logs (mentioned in #909) as an indicator for which common queries could be improved by the addition of indexes.
For Dexie.js, the current situation is a bit different. Because Dexie.js does not allow running queries on unindexed fields, every field is currently indexed. However, we can't always utilize the indexes because of some restrictions on how Dexie.js works. For example, Table.where
only accepts one index/one filter and cannot operate on multiple indexes at once. For queries with multiple filters, we currently run the query in-memory. This is not efficient for queries that return a large number of records.
There are a few steps we could take to improve Dexie.js performance as needed:
- For queries with more than one filter, we could use the first filter as the criteria for
Table.where
and then do further filtering in-memory. This is actually mentioned in a comment. - For common, performance-critical queries with multiple filters, we can in some cases use compound indexes to run the entire query without filtering in-memory. An example of one place where we already use this approach is the
[isNotPinned+expirationTimeSeconds]
index introduced in Re-implement dynamic max expiration time #832.