timetable-extension #1

Open

fred.boniface wants to merge 158 commits from timetable-extension into main

Author	SHA1	Message	Date
Fred Boniface	0f6f18b378	Add logging All checks were successful Go Test / test (push) Successful in 58s Details	2024-11-12 12:11:46 +00:00
Fred Boniface	5f57e1d267	Fix PIS inefficiencies and save output to file for debug purposes All checks were successful Go Test / test (push) Successful in 1m5s Details	2024-11-12 12:05:30 +00:00
Fred Boniface	0b3c1b6ad3	Temporarily fix issue finding TIPLOC for WPH All checks were successful Go Test / test (push) Successful in 1m33s Details	2024-11-12 11:59:23 +00:00
Fred Boniface	fc5bb2534a	Add additional logging to debug PIS update process All checks were successful Go Test / test (push) Successful in 2m36s Details	2024-11-11 11:03:25 +00:00
Fred Boniface	9b5779799f	Fix tar handling All checks were successful Go Test / test (push) Successful in 1m4s Details	2024-10-24 20:49:12 +01:00
Fred Boniface	28995790cc	Run PIS at startup All checks were successful Go Test / test (push) Successful in 1m24s Details	2024-10-24 20:39:49 +01:00
Fred Boniface	90bec49826	Adjustments to PIS Module	2024-10-24 20:39:24 +01:00
Fred Boniface	52732d2ae1	Adjust some VSTP logging	2024-10-24 11:36:45 +01:00
Fred Boniface	e15992e865	Adjust logging for testing All checks were successful Go Test / test (push) Successful in 2m19s Details	2024-10-24 10:42:27 +01:00
Fred Boniface	e8e715aa37	PIS module ready for testing All checks were successful Go Test / test (push) Successful in 1m52s Details	2024-10-23 11:42:47 +01:00
Fred Boniface	0320197147	Bump version All checks were successful Go Test / test (push) Successful in 2m17s Details	2024-10-22 20:51:20 +01:00
Fred Boniface	0881be7a79	Implement PIS task	2024-10-22 20:50:08 +01:00
Fred Boniface	8f0e9759c4	Add PIS Logic Some checks failed Go Test / test (push) Failing after 2m10s Details	2024-10-21 21:04:37 +01:00
Fred Boniface	9f86dd9e69	Add notes for next development step All checks were successful Go Test / test (push) Successful in 4m26s Details	2024-10-15 19:39:11 +01:00
Fred Boniface	f15aa27fdd	Bump version All checks were successful Go Test / test (push) Successful in 28s Details	2024-06-30 21:24:53 +01:00
Fred Boniface	77eb22b837	Testing new stations data All checks were successful Go Test / test (push) Successful in 26s Details	2024-06-30 20:28:34 +01:00
Fred Boniface	9496c9aae0	Add pipeline for tiploc/stanox All checks were successful Go Test / test (push) Successful in 31s Details	2024-06-30 19:36:37 +01:00
Fred Boniface	fed0208291	Complete alpha newStations data, deactivate CORPUS Stations data	2024-06-30 19:29:41 +01:00
Fred Boniface	358e69bec7	Add parsing for Stations All checks were successful Go Test / test (push) Successful in 31s Details	2024-06-30 09:45:16 +01:00
Fred Boniface	a153fd4990	Update test-db compose All checks were successful Go Test / test (push) Successful in 34s Details	2024-06-30 09:25:34 +01:00
Fred Boniface	e0e6c63f19	Update readme All checks were successful Go Test / test (push) Successful in 28s Details	2024-06-30 08:58:50 +01:00
Fred Boniface	7e55f2b635	Add compose for test-db All checks were successful Go Test / test (push) Successful in 1m11s Details	2024-06-30 08:56:56 +01:00
Fred Boniface	cd0f5c7003	Add stations modile All checks were successful Go Test / test (push) Successful in 26s Details	2024-06-29 21:54:32 +01:00
Fred Boniface	6baaba5033	Update dependencies All checks were successful Go Test / test (push) Successful in 1m17s Details	2024-06-28 15:08:37 +01:00
Fred Boniface	918ff1a151	Update go-types package All checks were successful Go Test / test (push) Successful in 1m3s Details	2024-06-28 14:38:20 +01:00
Fred Boniface	62387194d0	Prepare for stations All checks were successful Go Test / test (push) Successful in 2m4s Details	2024-06-28 13:59:58 +01:00
Fred Boniface	2cf567021d	Add coordinate types Some checks failed Go Test / test (push) Failing after 1m7s Details	2024-06-26 13:09:48 +01:00
Fred Boniface	e8e7a76076	Prep work for addition of location data for TIPLOCS Some checks failed Go Test / test (push) Has been cancelled Details	2024-06-26 13:07:44 +01:00
Fred Boniface	90e8fd5649	Begin PIS implementation All checks were successful Go Test / test (push) Successful in 2m3s Details	2024-05-02 15:35:00 +01:00
Fred Boniface	c5df223509	Add example DELETE message which needs work All checks were successful Go Test / test (push) Successful in 45s Details	2024-04-29 11:04:38 +01:00
Fred Boniface	5b1ab9b12f	Fix logging of failed VSTP messages All checks were successful Go Test / test (push) Successful in 43s Details	2024-04-28 11:52:15 +01:00
Fred Boniface	2adb82a1b3	Improve console fluff All checks were successful Go Test / test (push) Successful in 50s Details	2024-04-28 10:33:58 +01:00
Fred Boniface	b93d36dacd	Tidying error handling and adding some fluff. All checks were successful Go Test / test (push) Successful in 53s Details	2024-04-28 10:25:41 +01:00
Fred Boniface	01da611d26	Minor adjustments to test deployment All checks were successful Go Test / test (push) Successful in 3m43s Details	2024-04-28 08:38:53 +01:00
Fred Boniface	45b23b4cf0	Add VSTP message print when schedule segment is empty All checks were successful Go Test / test (push) Successful in 43s Details	2024-04-27 21:13:22 +01:00
Fred Boniface	ba6b6ccf40	Add debug files to dockerignore All checks were successful Go Test / test (push) Successful in 31s Details	2024-04-26 21:50:41 +01:00
Fred Boniface	b2beecc866	Bump version All checks were successful Go Test / test (push) Successful in 37s Details	2024-04-26 21:47:52 +01:00
Fred Boniface	7bae3575d5	Tidy VSTP logging All checks were successful Go Test / test (push) Successful in 52s Details	2024-04-26 20:48:17 +01:00
Fred Boniface	b2c9910856	Bump owlboard/go-types version All checks were successful Go Test / test (push) Successful in 1m2s Details	2024-04-24 23:19:47 +01:00
Fred Boniface	44adfea363	Add operator code to VSTP schedules	2024-04-24 23:19:34 +01:00
Fred Boniface	1231371bcf	Update owlboard/go-types All checks were successful Go Test / test (push) Successful in 1m3s Details	2024-04-24 23:09:53 +01:00
Fred Boniface	b0f9f547dd	Fix handling of VSTP messages	2024-04-24 23:09:34 +01:00
Fred Boniface	5b9c444ac5	Improve resiliency of STOMP subscription	2024-04-24 23:09:13 +01:00
Fred Boniface	f3745da86e	Improve readability	2024-04-24 23:08:51 +01:00
Fred Boniface	aef52be1e8	TransactionType not needed beyond processing stage. Removed from database.	2024-04-24 23:08:35 +01:00
Fred Boniface	9bd6bbde25	Prevent deleting VSTP services when completing full download. All checks were successful Go Test / test (push) Successful in 24s Details	2024-04-23 20:49:28 +01:00
Fred Boniface	a2c5bd49ff	Improve CIF Indexes	2024-04-23 20:10:04 +01:00
Fred Boniface	652d900618	Fix CIF Indexes All checks were successful Go Test / test (push) Successful in 31s Details	2024-04-23 16:54:29 +01:00
Fred Boniface	16bff981cc	Add logging for failed CIF index creation All checks were successful Go Test / test (push) Successful in 31s Details	2024-04-23 15:23:59 +01:00
Fred Boniface	9e65b66474	Ammend CIF Indexes All checks were successful Go Test / test (push) Successful in 34s Details	2024-04-23 15:20:49 +01:00
Fred Boniface	728e8f89e2	Add VSTP index All checks were successful Go Test / test (push) Successful in 41s Details	2024-04-23 15:13:16 +01:00
Fred Boniface	8e325339bf	Fix typing on generateServiceDetail function	2024-04-23 15:11:51 +01:00
Fred Boniface	e0a5bd86d7	Remove hasGuard option	2024-04-23 15:10:44 +01:00
Fred Boniface	73fd0063eb	Note adjustemnt	2024-04-23 15:10:20 +01:00
Fred Boniface	6cfddee1f3	Add notes All checks were successful Go Test / test (push) Successful in 26s Details	2024-04-23 00:32:03 +01:00
Fred Boniface	fe059abd6c	Adjust indexes on CIF collection All checks were successful Go Test / test (push) Successful in 24s Details	2024-04-23 00:31:03 +01:00
Fred Boniface	7bd9187e64	Add index creation All checks were successful Go Test / test (push) Successful in 25s Details	2024-04-23 00:27:33 +01:00
Fred Boniface	fd08fe3c4c	Update Dependencies All checks were successful Go Test / test (push) Successful in 59s Details	2024-04-22 12:30:31 +01:00
Fred Boniface	60ed218f07	Update CIF Conversion function to new ServiceDetail subdoc. All checks were successful Go Test / test (push) Successful in 2m24s Details	2024-04-22 10:58:00 +01:00
Fred Boniface	74e9b1b344	Fix daily download logic to fetch previous days update, rather than todays which is not the correct file. All checks were successful Go Test / test (push) Successful in 33s Details	2024-04-20 08:50:08 +01:00
Fred Boniface	db3d6030d5	Add debug logging to metadata.go All checks were successful Go Test / test (push) Successful in 48s Details	2024-04-20 08:44:14 +01:00
Fred Boniface	efd44da3ab	Add check for downloading old data. Prevent operating on old download All checks were successful Go Test / test (push) Successful in 1m35s Details	2024-04-19 20:38:11 +01:00
Fred Boniface	eefd9138de	Fix equation order for Cif Update duration All checks were successful Go Test / test (push) Successful in 43s Details	2024-04-17 09:40:15 +01:00
Fred Boniface	e890456390	Update owlboard/go-types All checks were successful Go Test / test (push) Successful in 1m50s Details	2024-04-16 21:31:33 +01:00
Fred Boniface	ff98adf1a6	Prevent running as root user All checks were successful Go Test / test (push) Successful in 1m5s Details	2024-04-16 19:20:19 +01:00
Fred Boniface	9ef9429511	Add duration log to CIF Update All checks were successful Go Test / test (push) Successful in 39s Details	2024-04-16 09:19:12 +01:00
Fred Boniface	790d293bc4	Update STOMP disconnection All checks were successful Go Test / test (push) Successful in 2m4s Details	2024-04-15 20:37:04 +01:00
Fred Boniface	ba8e4e4c72	Update VSTP package to use processing from CIF package	2024-04-15 20:37:04 +01:00
Fred Boniface	edbfbac23c	Include details about logging configuration Some checks failed Go Test / test (push) Failing after 1m13s Details	2024-04-14 22:59:58 +01:00
Fred Boniface	c86e601a62	Update dependencies Some checks failed Go Test / test (push) Failing after 2m10s Details	2024-04-14 22:51:31 +01:00
Fred Boniface	3d730054c0	Adjust log levels All checks were successful Go Test / test (push) Successful in 1m4s Details	2024-04-14 22:17:44 +01:00
Fred Boniface	be6300829d	Reogranise functions into appropriate files	2024-04-14 22:17:31 +01:00
Fred Boniface	421b68f936	Improve timezone handling	2024-04-14 22:17:01 +01:00
Fred Boniface	76f206441a	Tidy logging in package cif and dbAccess/cif.go	2024-04-14 21:28:17 +01:00
Fred Boniface	c9f894bbe6	Add RemoveOutdatedServices	2024-04-14 21:21:35 +01:00
Fred Boniface	f97bea78eb	Added multiple attempts with backoff delay to datebase connection	2024-04-14 21:21:13 +01:00
Fred Boniface	86da330b91	Tidied logging statements	2024-04-14 21:20:17 +01:00
Fred Boniface	c8709d67db	Log tidying	2024-04-14 20:07:56 +01:00
Fred Boniface	992005cc97	Optimise start-up and shut-down	2024-04-14 20:00:34 +01:00
Fred Boniface	91fd38104c	Reorganise logging	2024-04-14 19:03:13 +01:00
Fred Boniface	77dc11a658	Add additional logging to identify why CIF updates were sometimes failing All checks were successful Go Test / test (push) Successful in 46s Details	2024-04-13 21:45:24 +01:00
Fred Boniface	16fa1c5354	Adjust how performance logging is managed	2024-04-13 21:45:05 +01:00
Fred Boniface	dc958ac717	Set parsed to nil before exiting function	2024-04-13 21:44:53 +01:00
Fred Boniface	6ec8cd8895	Complete CIF Metadata parsing logic All checks were successful Go Test / test (push) Successful in 1m3s Details	2024-04-12 20:43:03 +01:00
Fred Boniface	94f880bcde	Add comment description to cif/types.go All checks were successful Go Test / test (push) Successful in 30s Details	2024-04-11 21:06:23 +01:00
Fred Boniface	251a190558	Set unordered for bulk CIF deletions from DB & reorganise dbAccess module.	2024-04-11 21:06:10 +01:00
Fred Boniface	76ae25fab6	Add LastUpdateType to CIF Metadata All checks were successful Go Test / test (push) Successful in 37s Details	2024-04-11 20:59:02 +01:00
Fred Boniface	4459d4d316	Update `package corpus` to stream data to the parse function. All checks were successful Go Test / test (push) Successful in 1m42s Details	2024-04-11 20:50:36 +01:00
Fred Boniface	b2f82b0250	Optimise batch size and stream handling	2024-04-11 20:50:13 +01:00
Fred Boniface	fb510e1408	Add logging for unknown CIF data types All checks were successful Go Test / test (push) Successful in 23s Details	2024-04-10 23:05:19 +01:00
Fred Boniface	751b67e61e	Remove unused functions All checks were successful Go Test / test (push) Successful in 25s Details	2024-04-10 22:59:13 +01:00
Fred Boniface	fea7a5b831	Disable parsing of JSONAssociations as they are not currently used within OwlBoard All checks were successful Go Test / test (push) Successful in 26s Details	2024-04-10 21:47:08 +01:00
Fred Boniface	3883031e04	Increase use of pointers in CORPUS package All checks were successful Go Test / test (push) Successful in 1m4s Details	2024-04-10 20:56:13 +01:00
Fred Boniface	39405e6d6a	Add comments	2024-04-10 20:46:20 +01:00
Fred Boniface	f903219276	Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable.	2024-04-10 20:19:16 +01:00
Fred Boniface	7acae49812	Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary All checks were successful Go Test / test (push) Successful in 31s Details	2024-04-09 22:39:35 +01:00
Fred Boniface	51f5c4fdcd	Adjust file write functions to reduce memory load	2024-04-09 22:38:48 +01:00
Fred Boniface	94e4cd964d	Improve CIF Parsing to acheive 60% memory reduction	2024-04-09 21:26:56 +01:00
Fred Boniface	a2c52f7b8b	Increase use of pointers to improve memory usage. Currently uses around 16GB RAM.	2024-04-09 21:07:21 +01:00
Fred Boniface	259f514b3d	Add function to write full CIF download to filesystem All checks were successful Go Test / test (push) Successful in 37s Details	2024-04-09 20:55:26 +01:00
Fred Boniface	2c34c529e1	Add debug option to write CIF data to file. All checks were successful Go Test / test (push) Successful in 1m20s Details	2024-04-09 20:38:40 +01:00
Fred Boniface	19f180d711	Expand CIF updates to ensure new sequence number is equal to oldSequence +1 All checks were successful Go Test / test (push) Successful in 32s Details	2024-04-08 22:13:08 +01:00
Fred Boniface	f243fc6831	Add debug heap memory allocation All checks were successful Go Test / test (push) Successful in 1m49s Details	2024-04-08 21:39:52 +01:00
Fred Boniface	7146d1a883	Add printing of number of goroutines to assist with performance improvements	2024-04-08 21:22:56 +01:00
Fred Boniface	3e5ed2c10a	Writing full CIF data now works. Still need to work on updating CIF data.	2024-04-08 21:08:07 +01:00
Fred Boniface	d9c249a47a	Remove development comments	2024-04-08 19:33:08 +01:00
Fred Boniface	feb7ee0fdf	Add test for parseStops	2024-04-08 19:33:00 +01:00
Fred Boniface	04a5600ca6	Update go-types	2024-04-08 19:20:23 +01:00
Fred Boniface	c4d680116a	Add additional keys for database.Service struct All checks were successful Go Test / test (push) Successful in 30s Details	2024-04-08 11:44:36 +01:00
Fred Boniface	d0c9250afa	Add notes All checks were successful Go Test / test (push) Successful in 1m38s Details	2024-04-07 21:29:09 +01:00
Fred Boniface	84b7d42898	Begin parsing of JsonSchedule	2024-04-07 21:28:48 +01:00
Fred Boniface	4a7bcd7f80	Increase use of pointers to minimise memory usage Some checks failed Go Test / test (push) Failing after 1m26s Details	2024-04-07 20:59:41 +01:00
Fred Boniface	3481c4e314	Tidy up All checks were successful Go Test / test (push) Successful in 1m43s Details	2024-04-06 22:31:38 +01:00
Fred Boniface	a15cbb778a	Implement generateMetadata function	2024-04-06 22:28:26 +01:00
Fred Boniface	9b53bbf5af	Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style.	2024-04-06 22:27:55 +01:00
Fred Boniface	ee2c8abd08	Add test for metadata generation	2024-04-06 22:27:14 +01:00
Fred Boniface	43d89119bf	Update dockerignore Some checks failed Go Test / test (push) Failing after 44s Details	2024-04-05 22:28:54 +01:00
Fred Boniface	426c5f02e8	Update dockerfile for new repo layout	2024-04-05 22:28:30 +01:00
Fred Boniface	493fb41da8	Reorganise repo Some checks failed Go Test / test (push) Failing after 48s Details	2024-04-05 22:23:42 +01:00
Fred Boniface	7bba8407d5	Add caching and go-version-file to action Some checks failed Go Test / test (push) Failing after 54s Details	2024-04-05 22:22:11 +01:00
Fred Boniface	78a8407f11	Try again to fix the test command! Some checks failed Go Test / test (push) Failing after 1m8s Details	2024-04-05 22:17:23 +01:00
Fred Boniface	eb0ca32158	Add cache env Some checks failed Go Test / test (push) Failing after 54s Details	2024-04-05 22:15:18 +01:00
Fred Boniface	c4e9a058af	Adjust test command to target src dir Some checks failed Go Test / test (push) Failing after 57s Details	2024-04-05 22:12:49 +01:00
Fred Boniface	6603fa979d	Fix test command (was go test ./.., now go test ./...) Some checks failed Go Test / test (push) Failing after 1m0s Details	2024-04-05 22:10:26 +01:00
Fred Boniface	0a7b399caf	Try again to get action working Some checks failed Go Test / test (push) Failing after 1m9s Details	2024-04-05 22:08:17 +01:00
Fred Boniface	2505c28794	Fix path for test running Some checks failed Go Test / test (push) Failing after 1m22s Details	2024-04-05 22:05:30 +01:00
Fred Boniface	872125f3b4	Fix action to run tests in correct directory Some checks failed Go Test / test (push) Failing after 1m44s Details	2024-04-05 22:01:03 +01:00
Fred Boniface	5f542664c9	Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion Some checks failed Go Test / test (push) Failing after 8m56s Details	2024-04-05 21:51:09 +01:00
Fred Boniface	3587b40703	Add action to run tests	2024-04-05 21:47:16 +01:00
Fred Boniface	e0edfd0d50	Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions	2024-04-05 21:42:00 +01:00
Fred Boniface	adf745aa61	Lots more work on CIF processing	2024-04-04 22:39:09 +01:00
Fred Boniface	2fdb840644	Add test to cif package	2024-04-04 13:58:18 +01:00
Fred Boniface	c8e6b9d7c8	Add comments for later help	2024-04-03 22:27:05 +01:00
Fred Boniface	3ebe7bfe11	Further work on CIF handling	2024-04-03 22:25:27 +01:00
Fred Boniface	2374a3ca37	Barely anything done because Lucy will not sleep.	2024-04-02 21:07:01 +01:00
Fred Boniface	3251b9a7bb	Update owlboard/go-types	2024-03-31 21:51:49 +01:00
Fred Boniface	12c37fe86d	Further work on CIF Data downloads	2024-03-30 01:09:12 +00:00
Fred Boniface	8a94e51437	Add comments to describe functions and types	2024-03-29 14:01:57 +00:00
Fred Boniface	2f5868e743	Implement shared logic between CORPUS and CIF packages. Begin implementation of shared logic between CIF and VSTP packages.	2024-03-29 13:45:58 +00:00
Fred Boniface	8c231fe4af	Remove unused const	2024-03-28 22:47:40 +00:00
Fred Boniface	d16634f07f	Add CIF checking logic	2024-03-28 22:47:08 +00:00
Fred Boniface	e76956bea2	Add notes for next steps	2024-03-27 23:38:39 +00:00
Fred Boniface	46c82eefa6	Add logic to avoid timetable updates too early	2024-03-27 23:35:03 +00:00
Fred Boniface	848f4bf7bd	Add dayString calculation for CIF updates	2024-03-27 23:08:51 +00:00
Fred Boniface	592675a24b	Complete CORPUS Update implementation and begin CIF update implementation	2024-03-26 22:33:11 +00:00
Fred Boniface	c144a98fd1	Add update time check function	2024-03-26 16:32:00 +00:00
Fred Boniface	74813e2fb0	Implement stations logic	2024-03-26 16:04:42 +00:00
Fred Boniface	f4a21ebe86	Integrate corpus fetch/parse logic	2024-03-26 15:40:15 +00:00
Fred Boniface	cf633eeb8f	Move ticker to new module	2024-03-26 11:44:19 +00:00
Fred Boniface	e204fb04a4	Adjust the program cleanup	2024-03-25 14:03:19 +00:00
Fred Boniface	d81cade73b	Refactor code to allow for centralised configuration fetching and the running of background ticker	2024-03-25 12:21:59 +00:00
Fred Boniface	b0cbab3e34	Change module name: - Update go target version - Change imports to reflect new module name - Update repo URL for new module name	2024-03-25 11:26:07 +00:00
Fred Boniface	57244c6971	Migrade from ioutil. to os for ReadFile	2024-03-25 10:51:29 +00:00
Fred Boniface	73c5509e9f	Implement new config_loader	2024-03-25 00:42:36 +00:00
Fred Boniface	4badddc9e6	Update Markdown format	2024-03-23 11:58:20 +00:00
Fred Boniface	a8258536e9	Adjust configuration table	2024-03-23 11:51:26 +00:00
Fred Boniface	77fd8c4c2d	Update logged references to `mq-client` and updated version number.	2024-03-23 11:34:52 +00:00
Fred Boniface	eef53ea2b6	Update readme	2024-03-23 11:26:24 +00:00

timetable-extension #1

158 Commits