Skip to main content

Using Web Scraping

Web scraping is nothing but the extraction of data from a website. Once the data is extracted, it can be stored for further analysis.

This type of automation might be essential to collect data and information, such as contacts, email addresses, stock data, cost matrix, telephone numbers, and any other search terms, from various websites. The extracted data can then be stored in a structured format either inside a database or a simple CSV or Excel file for further analysis.

Refer to list of Web Automation activities.

note

Currently, IntelliBuddies supports web automation using only the Chrome browser.

Pre-requisites​

  • Make sure URL to be used is open in Chrome before using element selector at design time. Refer to How to use Element selector for selecting appropriate element and extracting the required data.

  • Refer to simple example which explains how to scrap a text from web page and display the scrapped text using Message Box.

Currency Conversion Data Extraction Sample​

Let us understand the Web Scraping with one more example process.

Here is an example that takes input USD value and gets the conversion rate from the url and displays the final amount after conversion in INR.

Come, let's create this process in Process Designer:

Step#1​

Launch and create a new process in Process Designer

  • Launch Process Designer from the Windows Start menu located under IntelliBuddies Enterprise

Step#2​

Create New Process

  • Click on New Process and fill in the new process details

Step#3​

  • Add variables to hold the input and output details.

Step#4​

Step#5​

In Activity panel search for Input Dialog

  • Drag and drop Input Dialog activity into body of For Each.

  • Takes the input USD value and sets to int variable.

Step#6​

  • Create a sequence and add activities to get the XPath value from the config file containing a mapping of XPath value and string constant.

  • CSV and DataTable activities are used in Get_XPath_Value.xaml to get the appropriate XPath value from config.

  • Export this sequence as Snippet.

  • Drag and drop this Snippet in the main workflow, which adds invoke workflow activity, pointing to the above conversion workflow.

  • XPath config file

Step#7​

Step#8​

Step#9​

  • Add Writeline to display the converted value.

Step#10​

Save the process

  • From the Design tab ribbon, click on Save

Step#11​

Run the process

  • From the Design tab ribbon, click on Run

Step#12​

Make sure the process executes and pops up an input dialog. Enter the USD value to be converted to INR and click on Ok. Then in the output window, the converted value is displayed.

Download Example (C#)