|
|
76ae25fab6
|
Add LastUpdateType to CIF Metadata
Go Test / test (push) Successful in 37s
|
2024-04-11 20:59:02 +01:00 |
|
|
|
b2f82b0250
|
Optimise batch size and stream handling
|
2024-04-11 20:50:13 +01:00 |
|
|
|
39405e6d6a
|
Add comments
|
2024-04-10 20:46:20 +01:00 |
|
|
|
f903219276
|
Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable.
|
2024-04-10 20:19:16 +01:00 |
|
|
|
7acae49812
|
Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
Go Test / test (push) Successful in 31s
|
2024-04-09 22:39:35 +01:00 |
|
|
|
94e4cd964d
|
Improve CIF Parsing to acheive 60% memory reduction
|
2024-04-09 21:26:56 +01:00 |
|
|
|
a2c52f7b8b
|
Increase use of pointers to improve memory usage. Currently uses around 16GB RAM.
|
2024-04-09 21:07:21 +01:00 |
|
|
|
259f514b3d
|
Add function to write full CIF download to filesystem
Go Test / test (push) Successful in 37s
|
2024-04-09 20:55:26 +01:00 |
|
|
|
2c34c529e1
|
Add debug option to write CIF data to file.
Go Test / test (push) Successful in 1m20s
|
2024-04-09 20:38:40 +01:00 |
|
|
|
19f180d711
|
Expand CIF updates to ensure new sequence number is equal to oldSequence +1
Go Test / test (push) Successful in 32s
|
2024-04-08 22:13:08 +01:00 |
|
|
|
7146d1a883
|
Add printing of number of goroutines to assist with performance improvements
|
2024-04-08 21:22:56 +01:00 |
|
|
|
3e5ed2c10a
|
Writing full CIF data now works. Still need to work on updating CIF data.
|
2024-04-08 21:08:07 +01:00 |
|
|
|
3481c4e314
|
Tidy up
Go Test / test (push) Successful in 1m43s
|
2024-04-06 22:31:38 +01:00 |
|
|
|
a15cbb778a
|
Implement generateMetadata function
|
2024-04-06 22:28:26 +01:00 |
|
|
|
493fb41da8
|
Reorganise repo
Go Test / test (push) Failing after 48s
|
2024-04-05 22:23:42 +01:00 |
|
