Commit Graph

142 Commits

Author SHA1 Message Date
e0a5bd86d7 Remove hasGuard option 2024-04-23 15:10:44 +01:00
73fd0063eb Note adjustemnt 2024-04-23 15:10:20 +01:00
6cfddee1f3 Add notes
All checks were successful
Go Test / test (push) Successful in 26s
2024-04-23 00:32:03 +01:00
fe059abd6c Adjust indexes on CIF collection
All checks were successful
Go Test / test (push) Successful in 24s
2024-04-23 00:31:03 +01:00
7bd9187e64 Add index creation
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-23 00:27:33 +01:00
fd08fe3c4c Update Dependencies
All checks were successful
Go Test / test (push) Successful in 59s
2024-04-22 12:30:31 +01:00
60ed218f07 Update CIF Conversion function to new ServiceDetail subdoc.
All checks were successful
Go Test / test (push) Successful in 2m24s
2024-04-22 10:58:00 +01:00
74e9b1b344 Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
All checks were successful
Go Test / test (push) Successful in 33s
2024-04-20 08:50:08 +01:00
db3d6030d5 Add debug logging to metadata.go
All checks were successful
Go Test / test (push) Successful in 48s
2024-04-20 08:44:14 +01:00
efd44da3ab Add check for downloading old data. Prevent operating on old download
All checks were successful
Go Test / test (push) Successful in 1m35s
2024-04-19 20:38:11 +01:00
eefd9138de Fix equation order for Cif Update duration
All checks were successful
Go Test / test (push) Successful in 43s
2024-04-17 09:40:15 +01:00
e890456390 Update owlboard/go-types
All checks were successful
Go Test / test (push) Successful in 1m50s
2024-04-16 21:31:33 +01:00
ff98adf1a6 Prevent running as root user
All checks were successful
Go Test / test (push) Successful in 1m5s
2024-04-16 19:20:19 +01:00
9ef9429511 Add duration log to CIF Update
All checks were successful
Go Test / test (push) Successful in 39s
2024-04-16 09:19:12 +01:00
790d293bc4 Update STOMP disconnection
All checks were successful
Go Test / test (push) Successful in 2m4s
2024-04-15 20:37:04 +01:00
ba8e4e4c72 Update VSTP package to use processing from CIF package 2024-04-15 20:37:04 +01:00
edbfbac23c Include details about logging configuration
Some checks failed
Go Test / test (push) Failing after 1m13s
2024-04-14 22:59:58 +01:00
c86e601a62 Update dependencies
Some checks failed
Go Test / test (push) Failing after 2m10s
2024-04-14 22:51:31 +01:00
3d730054c0 Adjust log levels
All checks were successful
Go Test / test (push) Successful in 1m4s
2024-04-14 22:17:44 +01:00
be6300829d Reogranise functions into appropriate files 2024-04-14 22:17:31 +01:00
421b68f936 Improve timezone handling 2024-04-14 22:17:01 +01:00
76f206441a Tidy logging in package cif and dbAccess/cif.go 2024-04-14 21:28:17 +01:00
c9f894bbe6 Add RemoveOutdatedServices 2024-04-14 21:21:35 +01:00
f97bea78eb Added multiple attempts with backoff delay to datebase connection 2024-04-14 21:21:13 +01:00
86da330b91 Tidied logging statements 2024-04-14 21:20:17 +01:00
c8709d67db Log tidying 2024-04-14 20:07:56 +01:00
992005cc97 Optimise start-up and shut-down 2024-04-14 20:00:34 +01:00
91fd38104c Reorganise logging 2024-04-14 19:03:13 +01:00
77dc11a658 Add additional logging to identify why CIF updates were sometimes failing
All checks were successful
Go Test / test (push) Successful in 46s
2024-04-13 21:45:24 +01:00
16fa1c5354 Adjust how performance logging is managed 2024-04-13 21:45:05 +01:00
dc958ac717 Set parsed to nil before exiting function 2024-04-13 21:44:53 +01:00
6ec8cd8895 Complete CIF Metadata parsing logic
All checks were successful
Go Test / test (push) Successful in 1m3s
2024-04-12 20:43:03 +01:00
94f880bcde Add comment description to cif/types.go
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-11 21:06:23 +01:00
251a190558 Set unordered for bulk CIF deletions from DB & reorganise dbAccess module. 2024-04-11 21:06:10 +01:00
76ae25fab6 Add LastUpdateType to CIF Metadata
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-11 20:59:02 +01:00
4459d4d316 Update package corpus to stream data to the parse function.
All checks were successful
Go Test / test (push) Successful in 1m42s
2024-04-11 20:50:36 +01:00
b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
fb510e1408 Add logging for unknown CIF data types
All checks were successful
Go Test / test (push) Successful in 23s
2024-04-10 23:05:19 +01:00
751b67e61e Remove unused functions
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-10 22:59:13 +01:00
fea7a5b831 Disable parsing of JSONAssociations as they are not currently used within OwlBoard
All checks were successful
Go Test / test (push) Successful in 26s
2024-04-10 21:47:08 +01:00
3883031e04 Increase use of pointers in CORPUS package
All checks were successful
Go Test / test (push) Successful in 1m4s
2024-04-10 20:56:13 +01:00
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00