How to Automate Login to a Website to Download a File

Image

Computers are great at repeating mundane tasks so that users don't have to. Traditionally, though, you needed a computer science degree to make them do it. However, robotic process automation (RPA) makes it far easier to record and repeat those tasks, creating new opportunities for non-techies across the organization and taking the heat off the IT department.

Let's look at how to automate a login to a website and download a file. It's a task that users do manually every day to retrieve everything from procedure manuals to bug reports. We'll go through what it looks like with scripting using Python, JavaScript, C#, or PowerShell. And then we'll show you a tutorial for how to automate a website download without scripting.

Writing a Script to Login to a Website and Download a File

Traditionally, you'd automate this two-part process by writing a script using one of the popular languages available to developers, such as Python, Java, C#, or PowerShell.

There's a little work involved in using Python to log in to a website and download a file. The most obvious, but least successful way to do it is to use the Python requests library to access the URL in the login form's action attribute. That means manually reading the page source to find that URL and then passing it along with the values for the username and password through a requests.post instruction.

In Java, you can do something similar using the HttpURLConnection class, while C# coders might use the HttpWebRequest and HttpWebResponse classes. Those using PowerShell to log in and download a file might start by opening their browser as an object, using the Navigate action to reach the login URL, passing in the username and password, and then using click() to hit the login button.

These are basic login approaches, though, and prone to failure. Depending on the login form's structure, you might not be able to use them at all. For example, some sites will insist that your request includes url-encoded session tokens, leaving you with a 400 Bad Request response when you fail to pass them.

This is why many scripted attempts to automate website logins fail at the first hurdle.

Challenges With Scripted Website Automation

Password Storage and Retrieval

What do you type into the web form at login? Your username and password, naturally. That might seem simple at first glance, but where do you store those credentials? The most obvious place is in the script itself, but that's also the most insecure, because you'll be storing them in plain text. An alternative is to encode them in a separate encrypted file on the system, but then you'll need to decrypt that automatically. That means forcing the user to enter a decryption password.

All these passwords will be locally stored with no central management. What if you or the user want to update your password? What if the user leaves? With this scripting approach, you're on your own.

Scripts also get more complex if the site you're interacting with supports multi-factor authentication (MFA). While MFA is a great way to make a site more secure, it also means getting to grips with OAUTH, the token-based system typically used with MFA, introducing another layer of complexity into your script. 

Code Complexity

Then there's the problem of handling multiple outcomes. Scripts are great for this because you can use if-then or case statements to deal with multiple eventualities, but accommodating every single outcome increases the complexity of your code. A robust script must handle every kind of exception, from different response codes in HTTP headers through to login error messages.

All these considerations increase the complexity of your code, which grows as you address more web pages. Each page will have properties that require different code. This will force you to try and build some kind of master script that reads from a list of different parameters for each page, or more realistically to create an entirely new script for each new page that you visit. Either way, each time you add a new page to your list, you will have to tweak and debug the script. 

Maintainability

Then there's the problem of handling multiple outcomes. Scripts are great for this because you can use if-then or case statements to deal with multiple eventualities, but accommodating every single outcome increases the complexity of your code. A robust script must handle every kind of exception, from different response codes in HTTP headers through to login error messages.

All these considerations increase the complexity of your code, which grows as you address more web pages. Each page will have properties that require different code. This will force you to try and build some kind of master script that reads from a list of different parameters for each page, or more realistically to create an entirely new script for each new page that you visit. Either way, each time you add a new page to your list, you will have to tweak and debug the script. 

Code-Free Website Automation Tutorial

The alternative to scripting is to use an RPA tool that captures all these steps graphically. Automate provides a step recorder that allows for do-it-yourself UI-driven automation by capturing a user's interactions on browser-based or desktop applications. Each new action is a step that the user can edit at any time using simple dialog boxes without writing any code. Then the file download task can be used in larger workflows for file transfer automation.

Here's how you can use Automate to login to a website and download a file. Download a free Automate trial to follow along:

1. Users can begin by selecting an 'open web browser' action from the GUI Recorder menu and entering the URL that they want to surf to. The tool automatically waits for the entire page to load, and allows users to enter an optional wait time, just to be sure that everything has loaded before proceeding.
2. The tool follows the user’s interactions, recording clicks on individual fields in a form. The user can then enter his/her email address, select the password, and click the page's 'log in' or 'submit' button. The tool records these steps separately and does not use static screen coordinates in the process.
3. Unlike roll-your-own scripting solutions, Automate stores passwords in a secure central repository. It protects these passwords, so that not even administrators can view them, and only authorized users can access them.
4. Once logged into the site, the user can simply enter the URL for the file to download or mimic user navigation if necessary. When the browser reaches the file location, they can enter specific mouse clicks and keystrokes to simulate themselves issuing a 'save' command and can even open the file afterwards. The tool also offers scraper functionality that can parse tables, filenames, and link positions on a page.
5. Users can enter exception handling steps to deal with different header responses and take proactive action such as sending an email if any step of the automation fails, automatically incorporating informative exception information into the email.
6. After recording an automation task, users can schedule it to run via a simple workflow, which can contain other automation tasks or stand alone. It can run on any system in the network that the user has access to, and execution can be dynamically load balanced among various systems as well.

Advantages of No-Code Automation