Commit Graph

101 Commits

Author SHA1 Message Date
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
f243fc6831 Add debug heap memory allocation
All checks were successful
Go Test / test (push) Successful in 1m49s
2024-04-08 21:39:52 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
04a5600ca6 Update go-types 2024-04-08 19:20:23 +01:00
c4d680116a Add additional keys for database.Service struct
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-08 11:44:36 +01:00
d0c9250afa Add notes
All checks were successful
Go Test / test (push) Successful in 1m38s
2024-04-07 21:29:09 +01:00
84b7d42898 Begin parsing of JsonSchedule 2024-04-07 21:28:48 +01:00
4a7bcd7f80 Increase use of pointers to minimise memory usage
Some checks failed
Go Test / test (push) Failing after 1m26s
2024-04-07 20:59:41 +01:00
3481c4e314 Tidy up
All checks were successful
Go Test / test (push) Successful in 1m43s
2024-04-06 22:31:38 +01:00
a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
9b53bbf5af Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style. 2024-04-06 22:27:55 +01:00
ee2c8abd08 Add test for metadata generation 2024-04-06 22:27:14 +01:00
43d89119bf Update dockerignore
Some checks failed
Go Test / test (push) Failing after 44s
2024-04-05 22:28:54 +01:00
426c5f02e8 Update dockerfile for new repo layout 2024-04-05 22:28:30 +01:00
493fb41da8 Reorganise repo
Some checks failed
Go Test / test (push) Failing after 48s
2024-04-05 22:23:42 +01:00
7bba8407d5 Add caching and go-version-file to action
Some checks failed
Go Test / test (push) Failing after 54s
2024-04-05 22:22:11 +01:00
78a8407f11 Try again to fix the test command!
Some checks failed
Go Test / test (push) Failing after 1m8s
2024-04-05 22:17:23 +01:00
eb0ca32158 Add cache env
Some checks failed
Go Test / test (push) Failing after 54s
2024-04-05 22:15:18 +01:00
c4e9a058af Adjust test command to target src dir
Some checks failed
Go Test / test (push) Failing after 57s
2024-04-05 22:12:49 +01:00
6603fa979d Fix test command (was go test ./.., now go test ./...)
Some checks failed
Go Test / test (push) Failing after 1m0s
2024-04-05 22:10:26 +01:00
0a7b399caf Try again to get action working
Some checks failed
Go Test / test (push) Failing after 1m9s
2024-04-05 22:08:17 +01:00
2505c28794 Fix path for test running
Some checks failed
Go Test / test (push) Failing after 1m22s
2024-04-05 22:05:30 +01:00
872125f3b4 Fix action to run tests in correct directory
Some checks failed
Go Test / test (push) Failing after 1m44s
2024-04-05 22:01:03 +01:00
5f542664c9 Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion
Some checks failed
Go Test / test (push) Failing after 8m56s
2024-04-05 21:51:09 +01:00
3587b40703 Add action to run tests 2024-04-05 21:47:16 +01:00
e0edfd0d50 Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions 2024-04-05 21:42:00 +01:00
adf745aa61 Lots more work on CIF processing 2024-04-04 22:39:09 +01:00
2fdb840644 Add test to cif package 2024-04-04 13:58:18 +01:00
c8e6b9d7c8 Add comments for later help 2024-04-03 22:27:05 +01:00
3ebe7bfe11 Further work on CIF handling 2024-04-03 22:25:27 +01:00
2374a3ca37 Barely anything done because Lucy will not sleep. 2024-04-02 21:07:01 +01:00
3251b9a7bb Update owlboard/go-types 2024-03-31 21:51:49 +01:00
12c37fe86d Further work on CIF Data downloads 2024-03-30 01:09:12 +00:00
8a94e51437 Add comments to describe functions and types 2024-03-29 14:01:57 +00:00
2f5868e743 Implement shared logic between CORPUS and CIF packages.
Begin implementation of shared logic between CIF and VSTP packages.
2024-03-29 13:45:58 +00:00
8c231fe4af Remove unused const 2024-03-28 22:47:40 +00:00
d16634f07f Add CIF checking logic 2024-03-28 22:47:08 +00:00
e76956bea2 Add notes for next steps 2024-03-27 23:38:39 +00:00
46c82eefa6 Add logic to avoid timetable updates too early 2024-03-27 23:35:03 +00:00