Commit Graph

107 Commits

Author SHA1 Message Date
4459d4d316 Update package corpus to stream data to the parse function.
All checks were successful
Go Test / test (push) Successful in 1m42s
2024-04-11 20:50:36 +01:00
b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
fb510e1408 Add logging for unknown CIF data types
All checks were successful
Go Test / test (push) Successful in 23s
2024-04-10 23:05:19 +01:00
751b67e61e Remove unused functions
All checks were successful
Go Test / test (push) Successful in 25s
2024-04-10 22:59:13 +01:00
fea7a5b831 Disable parsing of JSONAssociations as they are not currently used within OwlBoard
All checks were successful
Go Test / test (push) Successful in 26s
2024-04-10 21:47:08 +01:00
3883031e04 Increase use of pointers in CORPUS package
All checks were successful
Go Test / test (push) Successful in 1m4s
2024-04-10 20:56:13 +01:00
39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
All checks were successful
Go Test / test (push) Successful in 31s
2024-04-09 22:39:35 +01:00
51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
259f514b3d Add function to write full CIF download to filesystem
All checks were successful
Go Test / test (push) Successful in 37s
2024-04-09 20:55:26 +01:00
2c34c529e1 Add debug option to write CIF data to file.
All checks were successful
Go Test / test (push) Successful in 1m20s
2024-04-09 20:38:40 +01:00
19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
All checks were successful
Go Test / test (push) Successful in 32s
2024-04-08 22:13:08 +01:00
f243fc6831 Add debug heap memory allocation
All checks were successful
Go Test / test (push) Successful in 1m49s
2024-04-08 21:39:52 +01:00
7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
04a5600ca6 Update go-types 2024-04-08 19:20:23 +01:00
c4d680116a Add additional keys for database.Service struct
All checks were successful
Go Test / test (push) Successful in 30s
2024-04-08 11:44:36 +01:00
d0c9250afa Add notes
All checks were successful
Go Test / test (push) Successful in 1m38s
2024-04-07 21:29:09 +01:00
84b7d42898 Begin parsing of JsonSchedule 2024-04-07 21:28:48 +01:00
4a7bcd7f80 Increase use of pointers to minimise memory usage
Some checks failed
Go Test / test (push) Failing after 1m26s
2024-04-07 20:59:41 +01:00
3481c4e314 Tidy up
All checks were successful
Go Test / test (push) Successful in 1m43s
2024-04-06 22:31:38 +01:00
a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
9b53bbf5af Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style. 2024-04-06 22:27:55 +01:00
ee2c8abd08 Add test for metadata generation 2024-04-06 22:27:14 +01:00
43d89119bf Update dockerignore
Some checks failed
Go Test / test (push) Failing after 44s
2024-04-05 22:28:54 +01:00
426c5f02e8 Update dockerfile for new repo layout 2024-04-05 22:28:30 +01:00
493fb41da8 Reorganise repo
Some checks failed
Go Test / test (push) Failing after 48s
2024-04-05 22:23:42 +01:00
7bba8407d5 Add caching and go-version-file to action
Some checks failed
Go Test / test (push) Failing after 54s
2024-04-05 22:22:11 +01:00
78a8407f11 Try again to fix the test command!
Some checks failed
Go Test / test (push) Failing after 1m8s
2024-04-05 22:17:23 +01:00
eb0ca32158 Add cache env
Some checks failed
Go Test / test (push) Failing after 54s
2024-04-05 22:15:18 +01:00
c4e9a058af Adjust test command to target src dir
Some checks failed
Go Test / test (push) Failing after 57s
2024-04-05 22:12:49 +01:00
6603fa979d Fix test command (was go test ./.., now go test ./...)
Some checks failed
Go Test / test (push) Failing after 1m0s
2024-04-05 22:10:26 +01:00
0a7b399caf Try again to get action working
Some checks failed
Go Test / test (push) Failing after 1m9s
2024-04-05 22:08:17 +01:00
2505c28794 Fix path for test running
Some checks failed
Go Test / test (push) Failing after 1m22s
2024-04-05 22:05:30 +01:00
872125f3b4 Fix action to run tests in correct directory
Some checks failed
Go Test / test (push) Failing after 1m44s
2024-04-05 22:01:03 +01:00
5f542664c9 Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion
Some checks failed
Go Test / test (push) Failing after 8m56s
2024-04-05 21:51:09 +01:00
3587b40703 Add action to run tests 2024-04-05 21:47:16 +01:00
e0edfd0d50 Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions 2024-04-05 21:42:00 +01:00
adf745aa61 Lots more work on CIF processing 2024-04-04 22:39:09 +01:00
2fdb840644 Add test to cif package 2024-04-04 13:58:18 +01:00
c8e6b9d7c8 Add comments for later help 2024-04-03 22:27:05 +01:00
3ebe7bfe11 Further work on CIF handling 2024-04-03 22:25:27 +01:00
2374a3ca37 Barely anything done because Lucy will not sleep. 2024-04-02 21:07:01 +01:00
3251b9a7bb Update owlboard/go-types 2024-03-31 21:51:49 +01:00
12c37fe86d Further work on CIF Data downloads 2024-03-30 01:09:12 +00:00