Fred Boniface
c9f894bbe6
|
||
|---|---|---|
| .. | ||
| check.go | ||
| constants.go | ||
| convert.go | ||
| convert_test.go | ||
| helpers.go | ||
| helpers_test.go | ||
| metadata.go | ||
| parse.go | ||
| process.go | ||
| process_test.go | ||
| readme.md | ||
| types.go | ||
| update.go | ||
readme.md
package cif
This package follows a similar pattern to package corpus.
First, CheckCorpus() retreived cif metadata from the database and determines whether an update is required and what type of update.
Then, one of the update functions is called which run through the update process. There are two update types, 'full' and 'update'. A 'full' update drops the entire timetable collection and rebuilds with a full CIF download. 'update' downloads CIF updates from specified days and applies updates.
Downloads are handled by package nrod which returns an io.ReadCloser which is passed to the parsing function.
Currently the parsing function returns a parsedCif pointer, however this is using significant memory due to the size of a full CIF download (Often around 4.5GB). The intention is to instead use a worker pool to handle the data.