Commit Graph

135 Commits

Author SHA1 Message Date
74e9b1b344 Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
All checks were successful
Go Test / test (push) Successful in 33s
2024-04-20 08:50:08 +01:00
db3d6030d5 Add debug logging to metadata.go
All checks were successful
Go Test / test (push) Successful in 48s
2024-04-20 08:44:14 +01:00
efd44da3ab Add check for downloading old data. Prevent operating on old download
All checks were successful
Go Test / test (push) Successful in 1m35s
2024-04-19 20:38:11 +01:00
eefd9138de Fix equation order for Cif Update duration
All checks were successful
Go Test / test (push) Successful in 43s
2024-04-17 09:40:15 +01:00
e890456390 Update owlboard/go-types
All checks were successful
Go Test / test (push) Successful in 1m50s
2024-04-16 21:31:33 +01:00
ff98adf1a6 Prevent running as root user
All checks were successful
Go Test / test (push) Successful in 1m5s
2024-04-16 19:20:19 +01:00
9ef9429511 Add duration log to CIF Update
All checks were successful
Go Test / test (push) Successful in 39s
2024-04-16 09:19:12 +01:00
790d293bc4 Update STOMP disconnection
All checks were successful
Go Test / test (push) Successful in 2m4s
2024-04-15 20:37:04 +01:00
ba8e4e4c72 Update VSTP package to use processing from CIF package 2024-04-15 20:37:04 +01:00
edbfbac23c Include details about logging configuration
Some checks failed
Go Test / test (push) Failing after 1m13s
2024-04-14 22:59:58 +01:00
c86e601a62 Update dependencies
Some checks failed
Go Test / test (push) Failing after 2m10s
2024-04-14 22:51:31 +01:00
3d730054c0 Adjust log levels
All checks were successful
Go Test / test (push) Successful in 1m4s
2024-04-14 22:17:44 +01:00
be6300829d Reogranise functions into appropriate files 2024-04-14 22:17:31 +01:00
421b68f936 Improve timezone handling 2024-04-14 22:17:01 +01:00
76f206441a Tidy logging in package cif and dbAccess/cif.go 2024-04-14 21:28:17 +01:00
c9f894bbe6 Add RemoveOutdatedServices 2024-04-14 21:21:35 +01:00
f97bea78eb Added multiple attempts with backoff delay to datebase connection 2024-04-14 21:21:13 +01:00
86da330b91 Tidied logging statements 2024-04-14 21:20:17 +01:00
c8709d67db Log tidying 2024-04-14 20:07:56 +01:00
992005cc97 Optimise start-up and shut-down 2024-04-14 20:00:34 +01:00
91fd38104c Reorganise logging 2024-04-14 19:03:13 +01:00
77dc11a658 Add additional logging to identify why CIF updates were sometimes failing
All checks were successful
Go Test / test (push) Successful in 46s
2024-04-13 21:45:24 +01:00
16fa1c5354 Adjust how performance logging is managed 2024-04-13 21:45:05 +01:00
dc958ac717 Set parsed to nil before exiting function 2024-04-13 21:44:53 +01:00
6ec8cd8895 Complete CIF Metadata parsing logic
All checks were successful
Go Test / test (push) Successful in 1m3s
2024-04-12 20:43:03 +01:00
94f880bcde Add comment description to cif/types.go
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-11 21:06:23 +01:00
251a190558 Set unordered for bulk CIF deletions from DB & reorganise dbAccess module. 2024-04-11 21:06:10 +01:00
76ae25fab6 Add LastUpdateType to CIF Metadata
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-11 20:59:02 +01:00
4459d4d316 Update package corpus to stream data to the parse function.
All checks were successful
Go Test / test (push) Successful in 1m42s
2024-04-11 20:50:36 +01:00
b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
fb510e1408 Add logging for unknown CIF data types
All checks were successful
Go Test / test (push) Successful in 23s
2024-04-10 23:05:19 +01:00
751b67e61e Remove unused functions
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-10 22:59:13 +01:00
fea7a5b831 Disable parsing of JSONAssociations as they are not currently used within OwlBoard
All checks were successful
Go Test / test (push) Successful in 26s
2024-04-10 21:47:08 +01:00
3883031e04 Increase use of pointers in CORPUS package
All checks were successful
Go Test / test (push) Successful in 1m4s
2024-04-10 20:56:13 +01:00
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
f243fc6831 Add debug heap memory allocation
All checks were successful
Go Test / test (push) Successful in 1m49s
2024-04-08 21:39:52 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
04a5600ca6 Update go-types 2024-04-08 19:20:23 +01:00
c4d680116a Add additional keys for database.Service struct
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-08 11:44:36 +01:00