|
|
b93d36dacd
|
Tidying error handling and adding some fluff.
Go Test / test (push) Successful in 53s
|
2024-04-28 10:25:41 +01:00 |
|
|
|
aef52be1e8
|
TransactionType not needed beyond processing stage. Removed from database.
|
2024-04-24 23:08:35 +01:00 |
|
|
|
9bd6bbde25
|
Prevent deleting VSTP services when completing full download.
Go Test / test (push) Successful in 24s
|
2024-04-23 20:49:28 +01:00 |
|
|
|
16bff981cc
|
Add logging for failed CIF index creation
Go Test / test (push) Successful in 31s
|
2024-04-23 15:23:59 +01:00 |
|
|
|
8e325339bf
|
Fix typing on generateServiceDetail function
|
2024-04-23 15:11:51 +01:00 |
|
|
|
e0a5bd86d7
|
Remove hasGuard option
|
2024-04-23 15:10:44 +01:00 |
|
|
|
73fd0063eb
|
Note adjustemnt
|
2024-04-23 15:10:20 +01:00 |
|
|
|
7bd9187e64
|
Add index creation
Go Test / test (push) Successful in 25s
|
2024-04-23 00:27:33 +01:00 |
|
|
|
60ed218f07
|
Update CIF Conversion function to new ServiceDetail subdoc.
Go Test / test (push) Successful in 2m24s
|
2024-04-22 10:58:00 +01:00 |
|
|
|
74e9b1b344
|
Fix daily download logic to fetch previous days update, rather than todays which is not the correct file.
Go Test / test (push) Successful in 33s
|
2024-04-20 08:50:08 +01:00 |
|
|
|
db3d6030d5
|
Add debug logging to metadata.go
Go Test / test (push) Successful in 48s
|
2024-04-20 08:44:14 +01:00 |
|
|
|
efd44da3ab
|
Add check for downloading old data. Prevent operating on old download
Go Test / test (push) Successful in 1m35s
|
2024-04-19 20:38:11 +01:00 |
|
|
|
eefd9138de
|
Fix equation order for Cif Update duration
Go Test / test (push) Successful in 43s
|
2024-04-17 09:40:15 +01:00 |
|
|
|
9ef9429511
|
Add duration log to CIF Update
Go Test / test (push) Successful in 39s
|
2024-04-16 09:19:12 +01:00 |
|
|
|
ba8e4e4c72
|
Update VSTP package to use processing from CIF package
|
2024-04-15 20:37:04 +01:00 |
|
|
|
be6300829d
|
Reogranise functions into appropriate files
|
2024-04-14 22:17:31 +01:00 |
|
|
|
421b68f936
|
Improve timezone handling
|
2024-04-14 22:17:01 +01:00 |
|
|
|
76f206441a
|
Tidy logging in package cif and dbAccess/cif.go
|
2024-04-14 21:28:17 +01:00 |
|
|
|
c9f894bbe6
|
Add RemoveOutdatedServices
|
2024-04-14 21:21:35 +01:00 |
|
|
|
c8709d67db
|
Log tidying
|
2024-04-14 20:07:56 +01:00 |
|
|
|
91fd38104c
|
Reorganise logging
|
2024-04-14 19:03:13 +01:00 |
|
|
|
77dc11a658
|
Add additional logging to identify why CIF updates were sometimes failing
Go Test / test (push) Successful in 46s
|
2024-04-13 21:45:24 +01:00 |
|
|
|
dc958ac717
|
Set parsed to nil before exiting function
|
2024-04-13 21:44:53 +01:00 |
|
|
|
6ec8cd8895
|
Complete CIF Metadata parsing logic
Go Test / test (push) Successful in 1m3s
|
2024-04-12 20:43:03 +01:00 |
|
|
|
94f880bcde
|
Add comment description to cif/types.go
Go Test / test (push) Successful in 30s
|
2024-04-11 21:06:23 +01:00 |
|
|
|
76ae25fab6
|
Add LastUpdateType to CIF Metadata
Go Test / test (push) Successful in 37s
|
2024-04-11 20:59:02 +01:00 |
|
|
|
b2f82b0250
|
Optimise batch size and stream handling
|
2024-04-11 20:50:13 +01:00 |
|
|
|
fb510e1408
|
Add logging for unknown CIF data types
Go Test / test (push) Successful in 23s
|
2024-04-10 23:05:19 +01:00 |
|
|
|
751b67e61e
|
Remove unused functions
Go Test / test (push) Successful in 25s
|
2024-04-10 22:59:13 +01:00 |
|
|
|
fea7a5b831
|
Disable parsing of JSONAssociations as they are not currently used within OwlBoard
Go Test / test (push) Successful in 26s
|
2024-04-10 21:47:08 +01:00 |
|
|
|
39405e6d6a
|
Add comments
|
2024-04-10 20:46:20 +01:00 |
|
|
|
f903219276
|
Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable.
|
2024-04-10 20:19:16 +01:00 |
|
|
|
7acae49812
|
Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
Go Test / test (push) Successful in 31s
|
2024-04-09 22:39:35 +01:00 |
|
|
|
51f5c4fdcd
|
Adjust file write functions to reduce memory load
|
2024-04-09 22:38:48 +01:00 |
|
|
|
94e4cd964d
|
Improve CIF Parsing to acheive 60% memory reduction
|
2024-04-09 21:26:56 +01:00 |
|
|
|
a2c52f7b8b
|
Increase use of pointers to improve memory usage. Currently uses around 16GB RAM.
|
2024-04-09 21:07:21 +01:00 |
|
|
|
259f514b3d
|
Add function to write full CIF download to filesystem
Go Test / test (push) Successful in 37s
|
2024-04-09 20:55:26 +01:00 |
|
|
|
2c34c529e1
|
Add debug option to write CIF data to file.
Go Test / test (push) Successful in 1m20s
|
2024-04-09 20:38:40 +01:00 |
|
|
|
19f180d711
|
Expand CIF updates to ensure new sequence number is equal to oldSequence +1
Go Test / test (push) Successful in 32s
|
2024-04-08 22:13:08 +01:00 |
|
|
|
7146d1a883
|
Add printing of number of goroutines to assist with performance improvements
|
2024-04-08 21:22:56 +01:00 |
|
|
|
3e5ed2c10a
|
Writing full CIF data now works. Still need to work on updating CIF data.
|
2024-04-08 21:08:07 +01:00 |
|
|
|
d9c249a47a
|
Remove development comments
|
2024-04-08 19:33:08 +01:00 |
|
|
|
feb7ee0fdf
|
Add test for parseStops
|
2024-04-08 19:33:00 +01:00 |
|
|
|
c4d680116a
|
Add additional keys for database.Service struct
Go Test / test (push) Successful in 30s
|
2024-04-08 11:44:36 +01:00 |
|
|
|
d0c9250afa
|
Add notes
Go Test / test (push) Successful in 1m38s
|
2024-04-07 21:29:09 +01:00 |
|
|
|
84b7d42898
|
Begin parsing of JsonSchedule
|
2024-04-07 21:28:48 +01:00 |
|
|
|
4a7bcd7f80
|
Increase use of pointers to minimise memory usage
Go Test / test (push) Failing after 1m26s
|
2024-04-07 20:59:41 +01:00 |
|
|
|
3481c4e314
|
Tidy up
Go Test / test (push) Successful in 1m43s
|
2024-04-06 22:31:38 +01:00 |
|
|
|
a15cbb778a
|
Implement generateMetadata function
|
2024-04-06 22:28:26 +01:00 |
|
|
|
ee2c8abd08
|
Add test for metadata generation
|
2024-04-06 22:27:14 +01:00 |
|
