OwlBoard
/
timetable-mgr
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

timetable-extension #1

Open
fred.boniface wants to merge 144 commits from timetable-extension into main
pull from: timetable-extension
merge into: OwlBoard:main
Conversation 0 Commits 144 Files Changed 66 +3446 -628

144 Commits

Author SHA1 Message Date
Fred Boniface f15aa27fdd Bump version
Go Test / test (push) Successful in 28s Details
2024-06-30 21:24:53 +01:00
Fred Boniface 77eb22b837 Testing new stations data
Go Test / test (push) Successful in 26s Details
2024-06-30 20:28:34 +01:00
Fred Boniface 9496c9aae0 Add pipeline for tiploc/stanox
Go Test / test (push) Successful in 31s Details
2024-06-30 19:36:37 +01:00
Fred Boniface fed0208291 Complete alpha newStations data, deactivate CORPUS Stations data 2024-06-30 19:29:41 +01:00
Fred Boniface 358e69bec7 Add parsing for Stations
Go Test / test (push) Successful in 31s Details
2024-06-30 09:45:16 +01:00
Fred Boniface a153fd4990 Update test-db compose
Go Test / test (push) Successful in 34s Details
2024-06-30 09:25:34 +01:00
Fred Boniface e0e6c63f19 Update readme
Go Test / test (push) Successful in 28s Details
2024-06-30 08:58:50 +01:00
Fred Boniface 7e55f2b635 Add compose for test-db
Go Test / test (push) Successful in 1m11s Details
2024-06-30 08:56:56 +01:00
Fred Boniface cd0f5c7003 Add stations modile
Go Test / test (push) Successful in 26s Details
2024-06-29 21:54:32 +01:00
Fred Boniface 6baaba5033 Update dependencies
Go Test / test (push) Successful in 1m17s Details
2024-06-28 15:08:37 +01:00
Fred Boniface 918ff1a151 Update go-types package
Go Test / test (push) Successful in 1m3s Details
2024-06-28 14:38:20 +01:00
Fred Boniface 62387194d0 Prepare for stations
Go Test / test (push) Successful in 2m4s Details
2024-06-28 13:59:58 +01:00
Fred Boniface 2cf567021d Add coordinate types
Go Test / test (push) Failing after 1m7s Details
2024-06-26 13:09:48 +01:00
Fred Boniface e8e7a76076 Prep work for addition of location data for TIPLOCS
Go Test / test (push) Has been cancelled Details
2024-06-26 13:07:44 +01:00
Fred Boniface 90e8fd5649 Begin PIS implementation
Go Test / test (push) Successful in 2m3s Details
2024-05-02 15:35:00 +01:00
Fred Boniface c5df223509 Add example DELETE message which needs work
Go Test / test (push) Successful in 45s Details
2024-04-29 11:04:38 +01:00
Fred Boniface 5b1ab9b12f Fix logging of failed VSTP messages
Go Test / test (push) Successful in 43s Details
2024-04-28 11:52:15 +01:00
Fred Boniface 2adb82a1b3 Improve console fluff
Go Test / test (push) Successful in 50s Details
2024-04-28 10:33:58 +01:00
Fred Boniface b93d36dacd Tidying error handling and adding some fluff.
Go Test / test (push) Successful in 53s Details
2024-04-28 10:25:41 +01:00
Fred Boniface 01da611d26 Minor adjustments to test deployment
Go Test / test (push) Successful in 3m43s Details
2024-04-28 08:38:53 +01:00
Fred Boniface 45b23b4cf0 Add VSTP message print when schedule segment is empty
Go Test / test (push) Successful in 43s Details
2024-04-27 21:13:22 +01:00
Fred Boniface ba6b6ccf40 Add debug files to dockerignore
Go Test / test (push) Successful in 31s Details
2024-04-26 21:50:41 +01:00
Fred Boniface b2beecc866 Bump version
Go Test / test (push) Successful in 37s Details
2024-04-26 21:47:52 +01:00
Fred Boniface 7bae3575d5 Tidy VSTP logging
Go Test / test (push) Successful in 52s Details
2024-04-26 20:48:17 +01:00
Fred Boniface b2c9910856 Bump owlboard/go-types version
Go Test / test (push) Successful in 1m2s Details
2024-04-24 23:19:47 +01:00
Fred Boniface 44adfea363 Add operator code to VSTP schedules 2024-04-24 23:19:34 +01:00
Fred Boniface 1231371bcf Update owlboard/go-types
Go Test / test (push) Successful in 1m3s Details
2024-04-24 23:09:53 +01:00
Fred Boniface b0f9f547dd Fix handling of VSTP messages 2024-04-24 23:09:34 +01:00
Fred Boniface 5b9c444ac5 Improve resiliency of STOMP subscription 2024-04-24 23:09:13 +01:00
Fred Boniface f3745da86e Improve readability 2024-04-24 23:08:51 +01:00
Fred Boniface aef52be1e8 TransactionType not needed beyond processing stage. Removed from database. 2024-04-24 23:08:35 +01:00
Fred Boniface 9bd6bbde25 Prevent deleting VSTP services when completing full download.
Go Test / test (push) Successful in 24s Details
2024-04-23 20:49:28 +01:00
Fred Boniface a2c5bd49ff Improve CIF Indexes 2024-04-23 20:10:04 +01:00
Fred Boniface 652d900618 Fix CIF Indexes
Go Test / test (push) Successful in 31s Details
2024-04-23 16:54:29 +01:00
Fred Boniface 16bff981cc Add logging for failed CIF index creation
Go Test / test (push) Successful in 31s Details
2024-04-23 15:23:59 +01:00
Fred Boniface 9e65b66474 Ammend CIF Indexes
Go Test / test (push) Successful in 34s Details
2024-04-23 15:20:49 +01:00
Fred Boniface 728e8f89e2 Add VSTP index
Go Test / test (push) Successful in 41s Details
2024-04-23 15:13:16 +01:00
Fred Boniface 8e325339bf Fix typing on generateServiceDetail function 2024-04-23 15:11:51 +01:00
Fred Boniface e0a5bd86d7 Remove hasGuard option 2024-04-23 15:10:44 +01:00
Fred Boniface 73fd0063eb Note adjustemnt 2024-04-23 15:10:20 +01:00
Fred Boniface 6cfddee1f3 Add notes
Go Test / test (push) Successful in 26s Details
2024-04-23 00:32:03 +01:00
Fred Boniface fe059abd6c Adjust indexes on CIF collection
Go Test / test (push) Successful in 24s Details
2024-04-23 00:31:03 +01:00
Fred Boniface 7bd9187e64 Add index creation
Go Test / test (push) Successful in 25s Details
2024-04-23 00:27:33 +01:00
Fred Boniface fd08fe3c4c Update Dependencies
Go Test / test (push) Successful in 59s Details
2024-04-22 12:30:31 +01:00
Fred Boniface 60ed218f07 Update CIF Conversion function to new ServiceDetail subdoc.
Go Test / test (push) Successful in 2m24s Details
2024-04-22 10:58:00 +01:00
Fred Boniface 74e9b1b344 Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
Go Test / test (push) Successful in 33s Details
2024-04-20 08:50:08 +01:00
Fred Boniface db3d6030d5 Add debug logging to metadata.go
Go Test / test (push) Successful in 48s Details
2024-04-20 08:44:14 +01:00
Fred Boniface efd44da3ab Add check for downloading old data. Prevent operating on old download
Go Test / test (push) Successful in 1m35s Details
2024-04-19 20:38:11 +01:00
Fred Boniface eefd9138de Fix equation order for Cif Update duration
Go Test / test (push) Successful in 43s Details
2024-04-17 09:40:15 +01:00
Fred Boniface e890456390 Update owlboard/go-types
Go Test / test (push) Successful in 1m50s Details
2024-04-16 21:31:33 +01:00
Fred Boniface ff98adf1a6 Prevent running as root user
Go Test / test (push) Successful in 1m5s Details
2024-04-16 19:20:19 +01:00
Fred Boniface 9ef9429511 Add duration log to CIF Update
Go Test / test (push) Successful in 39s Details
2024-04-16 09:19:12 +01:00
Fred Boniface 790d293bc4 Update STOMP disconnection
Go Test / test (push) Successful in 2m4s Details
2024-04-15 20:37:04 +01:00
Fred Boniface ba8e4e4c72 Update VSTP package to use processing from CIF package 2024-04-15 20:37:04 +01:00
Fred Boniface edbfbac23c Include details about logging configuration
Go Test / test (push) Failing after 1m13s Details
2024-04-14 22:59:58 +01:00
Fred Boniface c86e601a62 Update dependencies
Go Test / test (push) Failing after 2m10s Details
2024-04-14 22:51:31 +01:00
Fred Boniface 3d730054c0 Adjust log levels
Go Test / test (push) Successful in 1m4s Details
2024-04-14 22:17:44 +01:00
Fred Boniface be6300829d Reogranise functions into appropriate files 2024-04-14 22:17:31 +01:00
Fred Boniface 421b68f936 Improve timezone handling 2024-04-14 22:17:01 +01:00
Fred Boniface 76f206441a Tidy logging in package cif and dbAccess/cif.go 2024-04-14 21:28:17 +01:00
Fred Boniface c9f894bbe6 Add RemoveOutdatedServices 2024-04-14 21:21:35 +01:00
Fred Boniface f97bea78eb Added multiple attempts with backoff delay to datebase connection 2024-04-14 21:21:13 +01:00
Fred Boniface 86da330b91 Tidied logging statements 2024-04-14 21:20:17 +01:00
Fred Boniface c8709d67db Log tidying 2024-04-14 20:07:56 +01:00
Fred Boniface 992005cc97 Optimise start-up and shut-down 2024-04-14 20:00:34 +01:00
Fred Boniface 91fd38104c Reorganise logging 2024-04-14 19:03:13 +01:00
Fred Boniface 77dc11a658 Add additional logging to identify why CIF updates were sometimes failing
Go Test / test (push) Successful in 46s Details
2024-04-13 21:45:24 +01:00
Fred Boniface 16fa1c5354 Adjust how performance logging is managed 2024-04-13 21:45:05 +01:00
Fred Boniface dc958ac717 Set parsed to nil before exiting function 2024-04-13 21:44:53 +01:00
Fred Boniface 6ec8cd8895 Complete CIF Metadata parsing logic
Go Test / test (push) Successful in 1m3s Details
2024-04-12 20:43:03 +01:00
Fred Boniface 94f880bcde Add comment description to cif/types.go
Go Test / test (push) Successful in 30s Details
2024-04-11 21:06:23 +01:00
Fred Boniface 251a190558 Set unordered for bulk CIF deletions from DB & reorganise dbAccess module. 2024-04-11 21:06:10 +01:00
Fred Boniface 76ae25fab6 Add LastUpdateType to CIF Metadata
Go Test / test (push) Successful in 37s Details
2024-04-11 20:59:02 +01:00
Fred Boniface 4459d4d316 Update `package corpus` to stream data to the parse function.
Go Test / test (push) Successful in 1m42s Details
2024-04-11 20:50:36 +01:00
Fred Boniface b2f82b0250 Optimise batch size and stream handling 2024-04-11 20:50:13 +01:00
Fred Boniface fb510e1408 Add logging for unknown CIF data types
Go Test / test (push) Successful in 23s Details
2024-04-10 23:05:19 +01:00
Fred Boniface 751b67e61e Remove unused functions
Go Test / test (push) Successful in 25s Details
2024-04-10 22:59:13 +01:00
Fred Boniface fea7a5b831 Disable parsing of JSONAssociations as they are not currently used within OwlBoard
Go Test / test (push) Successful in 26s Details
2024-04-10 21:47:08 +01:00
Fred Boniface 3883031e04 Increase use of pointers in CORPUS package
Go Test / test (push) Successful in 1m4s Details
2024-04-10 20:56:13 +01:00
Fred Boniface 39405e6d6a Add comments 2024-04-10 20:46:20 +01:00
Fred Boniface f903219276 Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. 2024-04-10 20:19:16 +01:00
Fred Boniface 7acae49812 Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
Go Test / test (push) Successful in 31s Details
2024-04-09 22:39:35 +01:00
Fred Boniface 51f5c4fdcd Adjust file write functions to reduce memory load 2024-04-09 22:38:48 +01:00
Fred Boniface 94e4cd964d Improve CIF Parsing to acheive 60% memory reduction 2024-04-09 21:26:56 +01:00
Fred Boniface a2c52f7b8b Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. 2024-04-09 21:07:21 +01:00
Fred Boniface 259f514b3d Add function to write full CIF download to filesystem
Go Test / test (push) Successful in 37s Details
2024-04-09 20:55:26 +01:00
Fred Boniface 2c34c529e1 Add debug option to write CIF data to file.
Go Test / test (push) Successful in 1m20s Details
2024-04-09 20:38:40 +01:00
Fred Boniface 19f180d711 Expand CIF updates to ensure new sequence number is equal to oldSequence +1
Go Test / test (push) Successful in 32s Details
2024-04-08 22:13:08 +01:00
Fred Boniface f243fc6831 Add debug heap memory allocation
Go Test / test (push) Successful in 1m49s Details
2024-04-08 21:39:52 +01:00
Fred Boniface 7146d1a883 Add printing of number of goroutines to assist with performance improvements 2024-04-08 21:22:56 +01:00
Fred Boniface 3e5ed2c10a Writing full CIF data now works. Still need to work on updating CIF data. 2024-04-08 21:08:07 +01:00
Fred Boniface d9c249a47a Remove development comments 2024-04-08 19:33:08 +01:00
Fred Boniface feb7ee0fdf Add test for parseStops 2024-04-08 19:33:00 +01:00
Fred Boniface 04a5600ca6 Update go-types 2024-04-08 19:20:23 +01:00
Fred Boniface c4d680116a Add additional keys for database.Service struct
Go Test / test (push) Successful in 30s Details
2024-04-08 11:44:36 +01:00
Fred Boniface d0c9250afa Add notes
Go Test / test (push) Successful in 1m38s Details
2024-04-07 21:29:09 +01:00
Fred Boniface 84b7d42898 Begin parsing of JsonSchedule 2024-04-07 21:28:48 +01:00
Fred Boniface 4a7bcd7f80 Increase use of pointers to minimise memory usage
Go Test / test (push) Failing after 1m26s Details
2024-04-07 20:59:41 +01:00
Fred Boniface 3481c4e314 Tidy up
Go Test / test (push) Successful in 1m43s Details
2024-04-06 22:31:38 +01:00
Fred Boniface a15cbb778a Implement generateMetadata function 2024-04-06 22:28:26 +01:00
Fred Boniface 9b53bbf5af Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style. 2024-04-06 22:27:55 +01:00
Fred Boniface ee2c8abd08 Add test for metadata generation 2024-04-06 22:27:14 +01:00
Fred Boniface 43d89119bf Update dockerignore
Go Test / test (push) Failing after 44s Details
2024-04-05 22:28:54 +01:00
Fred Boniface 426c5f02e8 Update dockerfile for new repo layout 2024-04-05 22:28:30 +01:00
Fred Boniface 493fb41da8 Reorganise repo
Go Test / test (push) Failing after 48s Details
2024-04-05 22:23:42 +01:00
Fred Boniface 7bba8407d5 Add caching and go-version-file to action
Go Test / test (push) Failing after 54s Details
2024-04-05 22:22:11 +01:00
Fred Boniface 78a8407f11 Try again to fix the test command!
Go Test / test (push) Failing after 1m8s Details
2024-04-05 22:17:23 +01:00
Fred Boniface eb0ca32158 Add cache env
Go Test / test (push) Failing after 54s Details
2024-04-05 22:15:18 +01:00
Fred Boniface c4e9a058af Adjust test command to target src dir
Go Test / test (push) Failing after 57s Details
2024-04-05 22:12:49 +01:00
Fred Boniface 6603fa979d Fix test command (was go test ./.., now go test ./...)
Go Test / test (push) Failing after 1m0s Details
2024-04-05 22:10:26 +01:00
Fred Boniface 0a7b399caf Try again to get action working
Go Test / test (push) Failing after 1m9s Details
2024-04-05 22:08:17 +01:00
Fred Boniface 2505c28794 Fix path for test running
Go Test / test (push) Failing after 1m22s Details
2024-04-05 22:05:30 +01:00
Fred Boniface 872125f3b4 Fix action to run tests in correct directory
Go Test / test (push) Failing after 1m44s Details
2024-04-05 22:01:03 +01:00
Fred Boniface 5f542664c9 Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion
Go Test / test (push) Failing after 8m56s Details
2024-04-05 21:51:09 +01:00
Fred Boniface 3587b40703 Add action to run tests 2024-04-05 21:47:16 +01:00
Fred Boniface e0edfd0d50 Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions 2024-04-05 21:42:00 +01:00
Fred Boniface adf745aa61 Lots more work on CIF processing 2024-04-04 22:39:09 +01:00
Fred Boniface 2fdb840644 Add test to cif package 2024-04-04 13:58:18 +01:00
Fred Boniface c8e6b9d7c8 Add comments for later help 2024-04-03 22:27:05 +01:00
Fred Boniface 3ebe7bfe11 Further work on CIF handling 2024-04-03 22:25:27 +01:00
Fred Boniface 2374a3ca37 Barely anything done because Lucy will not sleep. 2024-04-02 21:07:01 +01:00
Fred Boniface 3251b9a7bb Update owlboard/go-types 2024-03-31 21:51:49 +01:00
Fred Boniface 12c37fe86d Further work on CIF Data downloads 2024-03-30 01:09:12 +00:00
Fred Boniface 8a94e51437 Add comments to describe functions and types 2024-03-29 14:01:57 +00:00
Fred Boniface 2f5868e743 Implement shared logic between CORPUS and CIF packages. 
Begin implementation of shared logic between CIF and VSTP packages.
 2024-03-29 13:45:58 +00:00
Fred Boniface 8c231fe4af Remove unused const 2024-03-28 22:47:40 +00:00
Fred Boniface d16634f07f Add CIF checking logic 2024-03-28 22:47:08 +00:00
Fred Boniface e76956bea2 Add notes for next steps 2024-03-27 23:38:39 +00:00
Fred Boniface 46c82eefa6 Add logic to avoid timetable updates too early 2024-03-27 23:35:03 +00:00
Fred Boniface 848f4bf7bd Add dayString calculation for CIF updates 2024-03-27 23:08:51 +00:00
Fred Boniface 592675a24b Complete CORPUS Update implementation and begin CIF update implementation 2024-03-26 22:33:11 +00:00
Fred Boniface c144a98fd1 Add update time check function 2024-03-26 16:32:00 +00:00
Fred Boniface 74813e2fb0 Implement stations logic 2024-03-26 16:04:42 +00:00
Fred Boniface f4a21ebe86 Integrate corpus fetch/parse logic 2024-03-26 15:40:15 +00:00
Fred Boniface cf633eeb8f Move ticker to new module 2024-03-26 11:44:19 +00:00
Fred Boniface e204fb04a4 Adjust the program cleanup 2024-03-25 14:03:19 +00:00
Fred Boniface d81cade73b Refactor code to allow for centralised configuration fetching and the running of background ticker 2024-03-25 12:21:59 +00:00
Fred Boniface b0cbab3e34 Change module name: 
- Update go target version
  - Change imports to reflect new module name
  - Update repo URL for new module name
 2024-03-25 11:26:07 +00:00
Fred Boniface 57244c6971 Migrade from ioutil. to os for ReadFile 2024-03-25 10:51:29 +00:00
Fred Boniface 73c5509e9f Implement new config_loader 2024-03-25 00:42:36 +00:00
Fred Boniface 4badddc9e6 Update Markdown format 2024-03-23 11:58:20 +00:00
Fred Boniface a8258536e9 Adjust configuration table 2024-03-23 11:51:26 +00:00
Fred Boniface 77fd8c4c2d Update logged references to `mq-client` and updated version number. 2024-03-23 11:34:52 +00:00
Fred Boniface eef53ea2b6 Update readme 2024-03-23 11:26:24 +00:00