 Fred Boniface
|
dc958ac717
|
Set parsed to nil before exiting function
|
2024-04-13 21:44:53 +01:00 |
|
Fred Boniface
|
6ec8cd8895
|
Complete CIF Metadata parsing logic
Go Test / test (push) Successful in 1m3s
Details
|
2024-04-12 20:43:03 +01:00 |
|
Fred Boniface
|
94f880bcde
|
Add comment description to cif/types.go
Go Test / test (push) Successful in 30s
Details
|
2024-04-11 21:06:23 +01:00 |
|
Fred Boniface
|
251a190558
|
Set unordered for bulk CIF deletions from DB & reorganise dbAccess module.
|
2024-04-11 21:06:10 +01:00 |
|
Fred Boniface
|
76ae25fab6
|
Add LastUpdateType to CIF Metadata
Go Test / test (push) Successful in 37s
Details
|
2024-04-11 20:59:02 +01:00 |
|
Fred Boniface
|
4459d4d316
|
Update `package corpus` to stream data to the parse function.
Go Test / test (push) Successful in 1m42s
Details
|
2024-04-11 20:50:36 +01:00 |
|
Fred Boniface
|
b2f82b0250
|
Optimise batch size and stream handling
|
2024-04-11 20:50:13 +01:00 |
|
Fred Boniface
|
fb510e1408
|
Add logging for unknown CIF data types
Go Test / test (push) Successful in 23s
Details
|
2024-04-10 23:05:19 +01:00 |
|
Fred Boniface
|
751b67e61e
|
Remove unused functions
Go Test / test (push) Successful in 25s
Details
|
2024-04-10 22:59:13 +01:00 |
|
Fred Boniface
|
fea7a5b831
|
Disable parsing of JSONAssociations as they are not currently used within OwlBoard
Go Test / test (push) Successful in 26s
Details
|
2024-04-10 21:47:08 +01:00 |
|
Fred Boniface
|
3883031e04
|
Increase use of pointers in CORPUS package
Go Test / test (push) Successful in 1m4s
Details
|
2024-04-10 20:56:13 +01:00 |
|
Fred Boniface
|
39405e6d6a
|
Add comments
|
2024-04-10 20:46:20 +01:00 |
|
Fred Boniface
|
f903219276
|
Streaming data into the CIF Parse function reduces memory down to 4.1GB - also looking to stream the parsed data directly into the database if feasable.
|
2024-04-10 20:19:16 +01:00 |
|
Fred Boniface
|
7acae49812
|
Re-implement processParsedCif() to reduce memory use by 10%. Further reductions are neccessary
Go Test / test (push) Successful in 31s
Details
|
2024-04-09 22:39:35 +01:00 |
|
Fred Boniface
|
51f5c4fdcd
|
Adjust file write functions to reduce memory load
|
2024-04-09 22:38:48 +01:00 |
|
Fred Boniface
|
94e4cd964d
|
Improve CIF Parsing to acheive 60% memory reduction
|
2024-04-09 21:26:56 +01:00 |
|
Fred Boniface
|
a2c52f7b8b
|
Increase use of pointers to improve memory usage. Currently uses around 16GB RAM.
|
2024-04-09 21:07:21 +01:00 |
|
Fred Boniface
|
259f514b3d
|
Add function to write full CIF download to filesystem
Go Test / test (push) Successful in 37s
Details
|
2024-04-09 20:55:26 +01:00 |
|
Fred Boniface
|
2c34c529e1
|
Add debug option to write CIF data to file.
Go Test / test (push) Successful in 1m20s
Details
|
2024-04-09 20:38:40 +01:00 |
|
Fred Boniface
|
19f180d711
|
Expand CIF updates to ensure new sequence number is equal to oldSequence +1
Go Test / test (push) Successful in 32s
Details
|
2024-04-08 22:13:08 +01:00 |
|
Fred Boniface
|
f243fc6831
|
Add debug heap memory allocation
Go Test / test (push) Successful in 1m49s
Details
|
2024-04-08 21:39:52 +01:00 |
|
Fred Boniface
|
7146d1a883
|
Add printing of number of goroutines to assist with performance improvements
|
2024-04-08 21:22:56 +01:00 |
|
Fred Boniface
|
3e5ed2c10a
|
Writing full CIF data now works. Still need to work on updating CIF data.
|
2024-04-08 21:08:07 +01:00 |
|
Fred Boniface
|
d9c249a47a
|
Remove development comments
|
2024-04-08 19:33:08 +01:00 |
|
Fred Boniface
|
feb7ee0fdf
|
Add test for parseStops
|
2024-04-08 19:33:00 +01:00 |
|
Fred Boniface
|
04a5600ca6
|
Update go-types
|
2024-04-08 19:20:23 +01:00 |
|
Fred Boniface
|
c4d680116a
|
Add additional keys for database.Service struct
Go Test / test (push) Successful in 30s
Details
|
2024-04-08 11:44:36 +01:00 |
|
Fred Boniface
|
d0c9250afa
|
Add notes
Go Test / test (push) Successful in 1m38s
Details
|
2024-04-07 21:29:09 +01:00 |
|
Fred Boniface
|
84b7d42898
|
Begin parsing of JsonSchedule
|
2024-04-07 21:28:48 +01:00 |
|
Fred Boniface
|
4a7bcd7f80
|
Increase use of pointers to minimise memory usage
Go Test / test (push) Failing after 1m26s
Details
|
2024-04-07 20:59:41 +01:00 |
|
Fred Boniface
|
3481c4e314
|
Tidy up
Go Test / test (push) Successful in 1m43s
Details
|
2024-04-06 22:31:38 +01:00 |
|
Fred Boniface
|
a15cbb778a
|
Implement generateMetadata function
|
2024-04-06 22:28:26 +01:00 |
|
Fred Boniface
|
9b53bbf5af
|
Implement function to insert new CIF entries in to the database and adjust the deletion function to follow the same style.
|
2024-04-06 22:27:55 +01:00 |
|
Fred Boniface
|
ee2c8abd08
|
Add test for metadata generation
|
2024-04-06 22:27:14 +01:00 |
|
Fred Boniface
|
43d89119bf
|
Update dockerignore
Go Test / test (push) Failing after 44s
Details
|
2024-04-05 22:28:54 +01:00 |
|
Fred Boniface
|
426c5f02e8
|
Update dockerfile for new repo layout
|
2024-04-05 22:28:30 +01:00 |
|
Fred Boniface
|
493fb41da8
|
Reorganise repo
Go Test / test (push) Failing after 48s
Details
|
2024-04-05 22:23:42 +01:00 |
|
Fred Boniface
|
7bba8407d5
|
Add caching and go-version-file to action
Go Test / test (push) Failing after 54s
Details
|
2024-04-05 22:22:11 +01:00 |
|
Fred Boniface
|
78a8407f11
|
Try again to fix the test command!
Go Test / test (push) Failing after 1m8s
Details
|
2024-04-05 22:17:23 +01:00 |
|
Fred Boniface
|
eb0ca32158
|
Add cache env
Go Test / test (push) Failing after 54s
Details
|
2024-04-05 22:15:18 +01:00 |
|
Fred Boniface
|
c4e9a058af
|
Adjust test command to target src dir
Go Test / test (push) Failing after 57s
Details
|
2024-04-05 22:12:49 +01:00 |
|
Fred Boniface
|
6603fa979d
|
Fix test command (was go test ./.., now go test ./...)
Go Test / test (push) Failing after 1m0s
Details
|
2024-04-05 22:10:26 +01:00 |
|
Fred Boniface
|
0a7b399caf
|
Try again to get action working
Go Test / test (push) Failing after 1m9s
Details
|
2024-04-05 22:08:17 +01:00 |
|
Fred Boniface
|
2505c28794
|
Fix path for test running
Go Test / test (push) Failing after 1m22s
Details
|
2024-04-05 22:05:30 +01:00 |
|
Fred Boniface
|
872125f3b4
|
Fix action to run tests in correct directory
Go Test / test (push) Failing after 1m44s
Details
|
2024-04-05 22:01:03 +01:00 |
|
Fred Boniface
|
5f542664c9
|
Add default binary names to .gitignore to prevent inflating repo size by accidental inclusion
Go Test / test (push) Failing after 8m56s
Details
|
2024-04-05 21:51:09 +01:00 |
|
Fred Boniface
|
3587b40703
|
Add action to run tests
|
2024-04-05 21:47:16 +01:00 |
|
Fred Boniface
|
e0edfd0d50
|
Port functions from VSTP to handle CIF data. Added tests for Cif/helpers functions
|
2024-04-05 21:42:00 +01:00 |
|
Fred Boniface
|
adf745aa61
|
Lots more work on CIF processing
|
2024-04-04 22:39:09 +01:00 |
|
Fred Boniface
|
2fdb840644
|
Add test to cif package
|
2024-04-04 13:58:18 +01:00 |