Commit Graph

26 Commits

Author SHA1 Message Date
9bd6bbde25 Prevent deleting VSTP services when completing full download.
All checks were successful
Go Test / test (push) Successful in 24s
2024-04-23 20:49:28 +01:00
16bff981cc Add logging for failed CIF index creation
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-23 15:23:59 +01:00
7bd9187e64 Add index creation
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-23 00:27:33 +01:00
74e9b1b344 Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
All checks were successful
Go Test / test (push) Successful in 33s
2024-04-20 08:50:08 +01:00
eefd9138de Fix equation order for Cif Update duration
All checks were successful
Go Test / test (push) Successful in 43s
2024-04-17 09:40:15 +01:00
9ef9429511 Add duration log to CIF Update
All checks were successful
Go Test / test (push) Successful in 39s
2024-04-16 09:19:12 +01:00
ba8e4e4c72 Update VSTP package to use processing from CIF package 2024-04-15 20:37:04 +01:00
c9f894bbe6 Add RemoveOutdatedServices 2024-04-14 21:21:35 +01:00
91fd38104c Reorganise logging 2024-04-14 19:03:13 +01:00
dc958ac717 Set parsed to nil before exiting function 2024-04-13 21:44:53 +01:00
6ec8cd8895 Complete CIF Metadata parsing logic
All checks were successful
Go Test / test (push) Successful in 1m3s
2024-04-12 20:43:03 +01:00
76ae25fab6 Add LastUpdateType to CIF Metadata
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-11 20:59:02 +01:00
b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
3481c4e314 Tidy up
All checks were successful
Go Test / test (push) Successful in 1m43s
2024-04-06 22:31:38 +01:00
a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
493fb41da8 Reorganise repo
Some checks failed
Go Test / test (push) Failing after 48s
2024-04-05 22:23:42 +01:00