Object | volume |
---|---|
Patients | 50.000.000 |
Visit | up to 40.000.000.000 |
Episode | up to 10.000.000.000 |
Encounter | up to 40.000.000.000 |
Observation | up to 500.000.000.000 |
Condition | up to 500.000.000.000 |
Allergy intolerance | 1.000.000.000 |
Immunization | 1.000.000.000 |
Submit encounter job is processed in stages:
# | Statement | Comment |
---|---|---|
1. | All the medical event POST/PUT operations should be async | Duration to process new records creation is not so critical for the medical service providers. They do have already all the data. Anyway it still should be processed in seconds/minutes, not hours/days. |
2. | GET operations will be sync | |
3. | Kafka will be used as a queue manager for the async operations |
|
4. | MongoDB will be used as a storage of the medical events data |
|
5. | Data validation and consistancy should be guaranteed on the application level | Application should ensure that operations will not be performed partially. |
6. | Medical events - new repository |
|
7. | Medical data can be accessed via API only by patient. There is no way to get the medical data of multiple patients via single call |
|
8. | We do split the medical data into three collections:
|
|
9. | We do split the storing data structure and presentation views (requests/responses) |
|
10. | We do guarantee jobs processing sequence for one patient_id. Not for all the jobs |
|
11. | ME database shouldn't store MPI_ID. Patient_id in ME will be encrypted using secret key. |
|
Database size: The MMAPv1 storage engine limits each database to no more than 16000 data files. This means that a single MMAPv1 database has a maximum size of 32TB. Setting the storage.mmapv1.smallFiles option reduces this limit to 8TB.
mongod
instance cannot manage a data set that exceeds maximum virtual memory address space provided by the underlying operating system. For Linux journaled data size of 64 TBCovered Queries in Sharded Clusters -
Starting in MongoDB 3.0, an index cannot cover a query on a sharded collection when run against a mongos
if the index does not contain the shard key, with the following exception for the _id
index: If a query on a sharded collection only specifies a condition on the _id
field and returns only the _id
field, the _id
index can cover the query when run against a mongos
even if the _id
field is not the shard key.A shard key cannot exceed 512 bytes.
Shard Key Index Type
A shard key index can be an ascending index on the shard key, a compound index that start with the shard key and specify ascending order for the shard key, or a hashed index.
A shard key index cannot be an index that specifies a multikey index, a text index or a geospatial index on the shard key fields.
# | Question | Answer |
---|---|---|
1. | Failed jobs processing | there are only two possible results of job processing: processed, error. Both of them mean that job has been successfully processed. |
2. | Jobs are processed FIFO. But it means that next job will not be processed until the current one is completed. It means that the whole system might block itself in case of problems of processing a single record. | in case if internal server or service error happens (50x), it will be forwarded to client. Option to be considered: put message back to the queue or move it to error queue. But in this case order processing cannot be guaranteed. |