timetable-extension #1

Open

fred.boniface wants to merge 160 commits from timetable-extension into main

pull from: timetable-extension

merge into: OwlBoard:main

OwlBoard:main

Conversation 0 Commits 160 Files Changed 67 +3832 -631

fred.boniface commented 2024-05-02 12:42:23 +01:00

Owner

Copy Link

Extend mq-client into timetable-mgr encompassing CIF updates, VSTP, CORPUS and PIS/Reason Codes. All of the elements that form the 'OwlBoard Timetable'

Extend mq-client into timetable-mgr encompassing CIF updates, VSTP, CORPUS and PIS/Reason Codes. All of the elements that form the 'OwlBoard Timetable'

fred.boniface added 129 commits 2024-05-02 12:42:39 +01:00

Update readme eef53ea2b6

Update logged references to mq-client and updated version number. 77fd8c4c2d

Adjust configuration table a8258536e9

Update Markdown format 4badddc9e6

Implement new config_loader 73c5509e9f

Migrade from ioutil. to os for ReadFile 57244c6971

Change module name: ... b0cbab3e34

- Update go target version
  - Change imports to reflect new module name
  - Update repo URL for new module name

Refactor code to allow for centralised configuration fetching and the running of background ticker d81cade73b

Adjust the program cleanup e204fb04a4

Move ticker to new module cf633eeb8f

Integrate corpus fetch/parse logic f4a21ebe86

Implement stations logic 74813e2fb0

Add update time check function c144a98fd1

Complete CORPUS Update implementation and begin CIF update implementation 592675a24b

Add dayString calculation for CIF updates 848f4bf7bd

Add logic to avoid timetable updates too early 46c82eefa6

Add notes for next steps e76956bea2

Add CIF checking logic d16634f07f

Remove unused const 8c231fe4af

Implement shared logic between CORPUS and CIF packages. ... 2f5868e743

Begin implementation of shared logic between CIF and VSTP packages.

Add comments to describe functions and types 8a94e51437

Further work on CIF Data downloads 12c37fe86d

Update owlboard/go-types 3251b9a7bb

Barely anything done because Lucy will not sleep. 2374a3ca37

Further work on CIF handling 3ebe7bfe11

Add comments for later help c8e6b9d7c8

Add test to cif package 2fdb840644

Lots more work on CIF processing adf745aa61

Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions e0edfd0d50

Add action to run tests 3587b40703

Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion 5f542664c9

Fix action to run tests in correct directory 872125f3b4

Fix path for test running 2505c28794

Try again to get action working 0a7b399caf

Fix test command (was go test ./.., now go test ./...) 6603fa979d

Adjust test command to target src dir c4e9a058af

Add cache env eb0ca32158

Try again to fix the test command! 78a8407f11

Add caching and go-version-file to action 7bba8407d5

Reorganise repo 493fb41da8

Update dockerfile for new repo layout 426c5f02e8

Update dockerignore 43d89119bf

Add test for metadata generation ee2c8abd08

Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style. 9b53bbf5af

Implement generateMetadata function a15cbb778a

Tidy up 3481c4e314

Increase use of pointers to minimise memory usage 4a7bcd7f80

Begin parsing of JsonSchedule 84b7d42898

Add notes d0c9250afa

Add additional keys for database.Service struct c4d680116a

Update go-types 04a5600ca6

Add test for parseStops feb7ee0fdf

Remove development comments d9c249a47a

Writing full CIF data now works. Still need to work on updating CIF data. 3e5ed2c10a

Add printing of number of goroutines to assist with performance improvements 7146d1a883

Add debug heap memory allocation f243fc6831

Expand CIF updates to ensure new sequence number is equal to oldSequence +1 19f180d711

Add debug option to write CIF data to file. 2c34c529e1

Add function to write full CIF download to filesystem 259f514b3d

Increase use of pointers to improve memory usage. Currently uses around 16GB RAM. a2c52f7b8b

Improve CIF Parsing to acheive 60% memory reduction 94e4cd964d

Adjust file write functions to reduce memory load 51f5c4fdcd

Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary 7acae49812

Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable. f903219276

Add comments 39405e6d6a

Increase use of pointers in CORPUS package 3883031e04

Disable parsing of JSONAssociations as they are not currently used within OwlBoard fea7a5b831

Remove unused functions 751b67e61e

Add logging for unknown CIF data types fb510e1408

Optimise batch size and stream handling b2f82b0250

Update package corpus to stream data to the parse function. 4459d4d316

Add LastUpdateType to CIF Metadata 76ae25fab6

Set unordered for bulk CIF deletions from DB & reorganise dbAccess module. 251a190558

Add comment description to cif/types.go 94f880bcde

Complete CIF Metadata parsing logic 6ec8cd8895

Set parsed to nil before exiting function dc958ac717

Adjust how performance logging is managed 16fa1c5354

Add additional logging to identify why CIF updates were sometimes failing 77dc11a658

Reorganise logging 91fd38104c

Optimise start-up and shut-down 992005cc97

Log tidying c8709d67db

Tidied logging statements 86da330b91

Added multiple attempts with backoff delay to datebase connection f97bea78eb

Add RemoveOutdatedServices c9f894bbe6

Tidy logging in package cif and dbAccess/cif.go 76f206441a

Improve timezone handling 421b68f936

Reogranise functions into appropriate files be6300829d

Adjust log levels 3d730054c0

Update dependencies c86e601a62

Include details about logging configuration edbfbac23c

Update VSTP package to use processing from CIF package ba8e4e4c72

Update STOMP disconnection 790d293bc4

Add duration log to CIF Update 9ef9429511

Prevent running as root user ff98adf1a6

Update owlboard/go-types e890456390

Fix equation order for Cif Update duration eefd9138de

Add check for downloading old data. Prevent operating on old download efd44da3ab

Add debug logging to metadata.go db3d6030d5

Fix daily download logic to fetch previous days update, rather than todays which is not the correct file. 74e9b1b344

Update CIF Conversion function to new ServiceDetail subdoc. 60ed218f07

Update Dependencies fd08fe3c4c

Add index creation 7bd9187e64

Adjust indexes on CIF collection fe059abd6c

Add notes 6cfddee1f3

Note adjustemnt 73fd0063eb

Remove hasGuard option e0a5bd86d7

Fix typing on generateServiceDetail function 8e325339bf

Add VSTP index 728e8f89e2

Ammend CIF Indexes 9e65b66474

Add logging for failed CIF index creation 16bff981cc

Fix CIF Indexes 652d900618

Improve CIF Indexes a2c5bd49ff

Prevent deleting VSTP services when completing full download. 9bd6bbde25

TransactionType not needed beyond processing stage. Removed from database. aef52be1e8

Improve readability f3745da86e

Improve resiliency of STOMP subscription 5b9c444ac5

Fix handling of VSTP messages b0f9f547dd

Update owlboard/go-types 1231371bcf

Add operator code to VSTP schedules 44adfea363

Bump owlboard/go-types version b2c9910856

Tidy VSTP logging 7bae3575d5

Bump version b2beecc866

Add debug files to dockerignore ba6b6ccf40

Add VSTP message print when schedule segment is empty 45b23b4cf0

Minor adjustments to test deployment 01da611d26

Tidying error handling and adding some fluff. b93d36dacd

Improve console fluff 2adb82a1b3

Fix logging of failed VSTP messages 5b1ab9b12f

Add example DELETE message which needs work c5df223509

fred.boniface added 1 commit 2024-05-02 15:35:13 +01:00

Begin PIS implementation 90e8fd5649

fred.boniface added 1 commit 2024-06-26 13:07:57 +01:00

Prep work for addition of location data for TIPLOCS e8e7a76076

fred.boniface added 1 commit 2024-06-26 13:09:58 +01:00

Add coordinate types 2cf567021d

fred.boniface added 1 commit 2024-06-28 14:00:10 +01:00

Prepare for stations 62387194d0

fred.boniface added 1 commit 2024-06-28 14:38:29 +01:00

Update go-types package 918ff1a151

fred.boniface added 1 commit 2024-06-28 15:08:47 +01:00

Update dependencies 6baaba5033

fred.boniface added 1 commit 2024-06-29 21:54:41 +01:00

Add stations modile cd0f5c7003

fred.boniface added 1 commit 2024-06-30 08:57:07 +01:00

Add compose for test-db 7e55f2b635

fred.boniface added 1 commit 2024-06-30 08:58:58 +01:00

Update readme e0e6c63f19

fred.boniface added 1 commit 2024-06-30 09:25:43 +01:00

Update test-db compose a153fd4990

fred.boniface added 1 commit 2024-06-30 09:45:23 +01:00

Add parsing for Stations 358e69bec7

fred.boniface added 2 commits 2024-06-30 19:36:46 +01:00

Complete alpha newStations data, deactivate CORPUS Stations data fed0208291

Add pipeline for tiploc/stanox 9496c9aae0

fred.boniface added 1 commit 2024-06-30 20:28:44 +01:00

Testing new stations data 77eb22b837

fred.boniface added 1 commit 2024-06-30 21:25:02 +01:00

Bump version f15aa27fdd

fred.boniface added 1 commit 2024-10-15 19:39:25 +01:00

Add notes for next development step 9f86dd9e69

fred.boniface added 1 commit 2024-10-21 21:04:46 +01:00

Add PIS Logic 8f0e9759c4

fred.boniface added 2 commits 2024-10-22 20:51:31 +01:00

Implement PIS task 0881be7a79

Bump version 0320197147

fred.boniface added 1 commit 2024-10-23 11:42:56 +01:00

PIS module ready for testing e8e715aa37

fred.boniface added 1 commit 2024-10-24 10:42:33 +01:00

Adjust logging for testing e15992e865

fred.boniface added 3 commits 2024-10-24 20:40:32 +01:00

Adjust some VSTP logging 52732d2ae1

Adjustments to PIS Module 90bec49826

Run PIS at startup 28995790cc

fred.boniface added 1 commit 2024-10-24 20:49:20 +01:00

Fix tar handling 9b5779799f

fred.boniface added 1 commit 2024-11-11 11:03:33 +00:00

Add additional logging to debug PIS update process fc5bb2534a

fred.boniface added 1 commit 2024-11-12 11:59:31 +00:00

Temporarily fix issue finding TIPLOC for WPH 0b3c1b6ad3

fred.boniface added 1 commit 2024-11-12 12:05:38 +00:00

Fix PIS inefficiencies and save output to file for debug purposes 5f57e1d267

fred.boniface added 1 commit 2024-11-12 12:11:53 +00:00

Add logging 0f6f18b378

fred.boniface added 1 commit 2024-11-13 00:28:18 +00:00

Temporarily deploy PIS data to testing collection d35a2a010e

fred.boniface added 1 commit 2024-11-13 10:22:50 +00:00

Fix PIS 'toc' and remove deduplication step 488551e914

All checks were successful

Hide all checks

Go Test / test (push) Successful in 58s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin timetable-extension:timetable-extension

git checkout timetable-extension

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No Label

Kind/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: OwlBoard/timetable-mgr#1

No description provided.