So I’ve been working lately on quite an overhaul to the InfoSquire backend. It involves the introduction of ‘Stateful Agent Executions’.
Essentially, it provides a mechanism for keeping track of exactly where an agent is in the exeuction process.. which URLs have been visited, which URLs have yet to be visited, how are URLs related (their path), which records have been saved, are yet to be saved or had problems being saved etc.. The whole state of the agent exeuction can at anytime be serialized and saved in case the execution needs to be stopped and resumed later. It also allows for a detailed analysis to be performed in the Agent debugger when a problem is reported.
In addition to these benefits, the stateful execution model will allow for the running of arbitrarily large or long-running agents since records will be saved to data repositories as they reach a ready-to-save state. Once records are saved, they can then be freed from memory.
The current stateless executions model is not as efficient for 3 reasons. 1) All records are kept in memory until agent exeuction is complete. 2) Large or long running agents can capture a lot of data and thus exhaust available memory, requiring multiple smaller agents to do the same work. 3) If something goes wrong during the execution process and causes it to crash, the process needs to start all over again.
Horray for stateful agents! This will be rolling into production over the next few weeks.
Comments
Add comment
Subscribe