Agent Works Only in Builder and not on Web Console

July 13, 2016

Mozenda agents are built and tested using a desktop computer. When an agent is processed, its actions are performed on Mozenda’s harvesting servers. These computers are more powerful than most desktop computers, and have been optimized for web scraping.

We do our best to run agents on these servers in a way that accurately reflects how the agent would behave on a standard desktop, but because of the varied factors involved in the way each unique website interacts with each unique computer, an agent’s behavior on our servers may not match its behavior in the Agent Builder.

This article is a compilation of common solutions for agents that work while testing in the Agent Builder but return an error in the Web Console. Because of the nuanced nature of this subject, the actual causes are varied. If the solutions in this article do not resolve the issues you are encountering, contact Mozenda Support for assistance.

This article is divided into the following sections:

  1. Adjust the Error Handling Settings
  2. Adjust the Advanced Settings
  3. Slow the Agent Down
  4. Navigation Style
  5. Syntax
  6. Collection and Field Settings
  7. Site Preferences

Adjust the Error Handling Settings

This setting is useful if the agent in question is collecting blank records where you know it should be collecting data, and no error is identified. By default, agents are set to attempt to ignore errors. This means that the system will ignore each error type a certain number of times. The system will pause the agent and show an error status only after a particular error is consecutively encountered by the same action too many times.

You can set the agent to stop immediately when it encounters an error, giving you the opportunity to refine the agent against that type of error in the future.

  1. Open the agent in the Web Console.
  2. Open the Tools menu and click Settings.
  3. Click the Error Handling tab.
  4. Open the drop-down menu under When an agent error occurs and click Stop the job so the agent can be fixed.
  5. Open the drop-down menu under When a website error occurs and click Stop the job so the agent can be fixed.
  6. Click Save.

Adjust the Advanced Settings

The Mozenda harvesters attempt to run every agent as quickly and efficiently as possible. By default, multiple items from the same list are processed at the same time. For more dynamic websites, this may cause unpredictable behavior.

You can make the agent more stable by setting it to process only one list item at a time.

  1. Open the agent in the Web Console.
  2. Open the Tools menu and click Settings.
  3. Click the Advanced tab.
  4. Check the box next to Do not process multiple list items concurrently.
  5. Click Save.

Slow the Agent Down

When an agent is set up in the Agent Builder, it runs on your local computer. When that agent is run in the Web Console, it runs on the Mozenda harvesting servers. These servers are more powerful than a standard desktop, and will generally perform actions faster than a personal computer. This can lead to the agent looking for data before the target objects have loaded on the web page.

Adding wait seconds to an agent can give the web page an opportunity to finish loading before the agent attempts to perform actions. It’s a good idea to do this for each page in the agent.

  1. Open the agent in the Agent Builder.
  2. Right-click the page title of a page of actions in the Agent Builder.
  3. Click After Loading This Page.
  4. In Wait up to 15 seconds for this page to load, increase the time from 15 to 30.

    This does not mean the agent will wait 30 seconds each time, but rather that the agent will wait a maximum of 30 seconds for the page to load.
  5. In Wait 0 seconds before performing the first action on this page, increase the time from 0 to 5.
  6. Check the box next to Wait for AJAX to alter this page when it is done loading.
  7. Click Save.

Pages That Load With AJAX

Some websites use AJAX and other dynamic-content-loading methods to change the page content instead of navigating to a new URL. When scraping these websites, it may be necessary to change the navigation style of the agent so that the agent waits for AJAX instead of loading a new page when a link is clicked. The result is an agent where most or all of the actions are on the same page of actions.

  1. Right-click the Click Item or Set User Input action that navigates from page 1 to page 2, and choose Properties.
  2. Click After Performing Action.
  3. Check the box next to Wait for AJAX to alter the current page.
  4. Click Save.
  5. Copy the actions from Page 2 and paste them onto Page 1 directly after the Click Item or Set User Input action.
  6. Delete Page 2.
  7. Repeat this process for all the click item actions in the agent until all the actions appear on the same page of actions in the order which they will be performed.

Order or Actions

Check the order of actions in the action list. Mozenda executes these actions in the order they appear in the action list. Here are a few things to look for:

  1. Ensure that you don’t have two Click Item actions going to different web pages on the same page of actions.

    This may not cause issues in the Agent Builder, but because of processing limitations, this will produce an error in the Web Console.

  2. Verify that any Click Item actions are always the last action listed in the Item List or on the page of actions.
    1. The only actions that should appear after a click item that navigates to a new page is an End List or Page List action.
    2. Click Item actions that wait for AJAX or do not navigate to a new page do not need to come at the end of the Action List.

Collection and Field Settings

If the agent has unique fields or contributes to a combined collection that has unique fields, some records may be dropped or ignored because of unique fields. Here are some things to look out for:

  1. Generally, do not make a unique field when it does not contain truly unique data.

    If a unique field encounters a record with data it has encountered before, it will drop the entire record.

  2. Confirm that the total number of characters from one or all unique fields added together equals less than 450. Otherwise, the record will be dropped.
  3. If an agent is part of a combined collection, confirm that the agent collection has the same unique fields that the combined collection has.

Site Preferences

The harvesters (servers) that Mozenda uses to run an agent are located in the U.S.A. If you are from a location outside the U.S.A. and you build an agent for a site that changes its content based on your location, this may cause problems. Some websites provide the option to manually select the location, which should be built into the agent when available. Other websites often include the location in the URL, which you can modify using the Load Page action.

Need more information?

We are anxious to answer any questions you may have about our products and services.

100% Privacy. You are that important to us. Privacy Policy

X