Commit Graph

42 Commits

Author SHA1 Message Date
74e9b1b344 Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
All checks were successful
Go Test / test (push) Successful in 33s
2024-04-20 08:50:08 +01:00
db3d6030d5 Add debug logging to metadata.go
All checks were successful
Go Test / test (push) Successful in 48s
2024-04-20 08:44:14 +01:00
efd44da3ab Add check for downloading old data. Prevent operating on old download
All checks were successful
Go Test / test (push) Successful in 1m35s
2024-04-19 20:38:11 +01:00
eefd9138de Fix equation order for Cif Update duration
All checks were successful
Go Test / test (push) Successful in 43s
2024-04-17 09:40:15 +01:00
9ef9429511 Add duration log to CIF Update
All checks were successful
Go Test / test (push) Successful in 39s
2024-04-16 09:19:12 +01:00
ba8e4e4c72 Update VSTP package to use processing from CIF package 2024-04-15 20:37:04 +01:00
be6300829d Reogranise functions into appropriate files 2024-04-14 22:17:31 +01:00
421b68f936 Improve timezone handling 2024-04-14 22:17:01 +01:00
76f206441a Tidy logging in package cif and dbAccess/cif.go 2024-04-14 21:28:17 +01:00
c9f894bbe6 Add RemoveOutdatedServices 2024-04-14 21:21:35 +01:00
c8709d67db Log tidying 2024-04-14 20:07:56 +01:00
91fd38104c Reorganise logging 2024-04-14 19:03:13 +01:00
77dc11a658 Add additional logging to identify why CIF updates were sometimes failing
All checks were successful
Go Test / test (push) Successful in 46s
2024-04-13 21:45:24 +01:00
dc958ac717 Set parsed to nil before exiting function 2024-04-13 21:44:53 +01:00
6ec8cd8895 Complete CIF Metadata parsing logic
All checks were successful
Go Test / test (push) Successful in 1m3s
2024-04-12 20:43:03 +01:00
94f880bcde Add comment description to cif/types.go
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-11 21:06:23 +01:00
76ae25fab6 Add LastUpdateType to CIF Metadata
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-11 20:59:02 +01:00
b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
fb510e1408 Add logging for unknown CIF data types
All checks were successful
Go Test / test (push) Successful in 23s
2024-04-10 23:05:19 +01:00
751b67e61e Remove unused functions
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-10 22:59:13 +01:00
fea7a5b831 Disable parsing of JSONAssociations as they are not currently used within OwlBoard
All checks were successful
Go Test / test (push) Successful in 26s
2024-04-10 21:47:08 +01:00
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
c4d680116a Add additional keys for database.Service struct
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-08 11:44:36 +01:00
d0c9250afa Add notes
All checks were successful
Go Test / test (push) Successful in 1m38s
2024-04-07 21:29:09 +01:00
84b7d42898 Begin parsing of JsonSchedule 2024-04-07 21:28:48 +01:00
4a7bcd7f80 Increase use of pointers to minimise memory usage
Some checks failed
Go Test / test (push) Failing after 1m26s
2024-04-07 20:59:41 +01:00
3481c4e314 Tidy up
All checks were successful
Go Test / test (push) Successful in 1m43s
2024-04-06 22:31:38 +01:00
a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
ee2c8abd08 Add test for metadata generation 2024-04-06 22:27:14 +01:00
493fb41da8 Reorganise repo
Some checks failed
Go Test / test (push) Failing after 48s
2024-04-05 22:23:42 +01:00