Commit Graph

21 Commits

Author SHA1 Message Date
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
c4d680116a Add additional keys for database.Service struct
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-08 11:44:36 +01:00
d0c9250afa Add notes
All checks were successful
Go Test / test (push) Successful in 1m38s
2024-04-07 21:29:09 +01:00
84b7d42898 Begin parsing of JsonSchedule 2024-04-07 21:28:48 +01:00
4a7bcd7f80 Increase use of pointers to minimise memory usage
Some checks failed
Go Test / test (push) Failing after 1m26s
2024-04-07 20:59:41 +01:00
3481c4e314 Tidy up
All checks were successful
Go Test / test (push) Successful in 1m43s
2024-04-06 22:31:38 +01:00
a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
ee2c8abd08 Add test for metadata generation 2024-04-06 22:27:14 +01:00
493fb41da8 Reorganise repo
Some checks failed
Go Test / test (push) Failing after 48s
2024-04-05 22:23:42 +01:00