Testing websites for performance problems while they are in a pre-production environment is the best way to ensure that fast sites stay fast. To help, Rigor’s web performance analysis product, Rigor Optimization supports many options to scan development and pre-production systems. Optimization also supports the ability to scan complex or single page applications via scripting. However, these are advanced features and configuring them can be tough and time-consuming. In this post, I’ll discuss a new feature, auditing HARs with full response bodies, that makes it much easier analyze any site for performance issues. Now, if you can see it in your browser, Optimization can scan it!

Traditional Options for Scanning Pre-production

Optimization can audit all aspects of your website, from just a single page to crawling the entire site. Underlying all of that is the requirement that our scanning systems can actually reach and communicate with the website you want to assess. For public systems, that’s not a problem. If you are trying to scan a staging or pre-production system, however, you often need to do more work for our scanners to be able to access these systems. Traditionally this means needing to whitelist the IPs of our scanners. We support other options like configuring our scanners to contacting your systems using HTTP Authentication or even client-side SSL certificates. You can even set custom HTTP headers or custom cookie values to help identify traffic from our scanner systems and allow it through.

As easy as we try and make it, all of those options still require work. You either have to do work to configure our Optimization tests in a certain way and potentially have to configure your pre-production environment to allow us to access the site. This is less than ideal.

Traditional Options Scanning Complex Applications

Optimization has supported the ability to scan complex web apps for quite some time via CasperJS scripts. CasperJS is a great way to write scripts to walk through various parts of your website, and it already has strong adoption in the functional testing space.

When using Rigor Optimization’s CasperJS option we execute the CasperJS script, capture all the content that gets loaded, and audit the content for performance issues. CasperJS support allows you to test multiple step processes like checking out or adding an item to a shopping cart, or even test different workflows inside of single page apps.

However, much like scanning pre-production systems, using CasperJS to scan a complex app still requires work. You must bring your CasperJS scripts over and configure Optimization tests to use them. If you aren’t using CasperJS, you need to create those scripts, or work with our Client Success team to create CasperJS scripts that give you the site coverage you need.

If you can see it, Optimization can scan it

While we have robust options that you can use to configure Optimization to analyze pre-production system or to walk through complex systems, we wanted to make this easier. The solution was to leverage the user’s browser. If a user could access an internal site from their browser, or use it to walk through a series of pages, why can’t Optimization just audit that content? In short, if I can see it, can Optimization scan it?

So that’s what we did.

I am pleased to announce that Optimization now supports HAR files that contain full response bodies. This means if you give us a HAR file which contains the full response bodies, Optimization won’t make any requests for content to audit. The content already exists in the HAR file, so Optimization just audits that content for performance issues. This lets you completely avoid having to configure Optimization to access the original website, or pre-production system, or complicated application flow.

How does that work? Well, all modern desktop web browsers allow you to record network traffic, and save the results as a HAR file with full response bodies! Basically, your browser acts as a VCR, recording everything we need about a site into a HAR file. When you upload that HAR file to Optimization, we audit all that content. If you can see it in your web browser, Optimization can audit it!

Capturing Full Response HARs in the Browser

Here is how it works. Start off by going to the URL you want Optimization to audit in your desktop browser. Then open the browser’s “developer tools.” On Chrome, you will find these tools under Menu icon > More Tools > Developer Tools. Now select the Network Tab.

net-tab-empty

 

If you are going to scan multiple pages, you will want to check Preserve Log. This tells the browser to keep all the data it sees, even across multiple pages. You should also check Disable Cache to force your browser to actually request all the content for the page, ensure you record it.

Now, reload the page. You’ll see the network panel fill up with all the page’s content. If you are testing multiple pages, navigate around. If you are working with a complex app, use the app as you want. The browser will record everything in that Network Panel, as shown below.

network- tab

When you are done capturing all the content that you want. Right click in the network view, and select Save as HAR with Content. Firefox has a similarly worded option.

export-har

Now you are ready to upload the HAR into Optimization. Go to Create a New Test > click the Advanced tab > select HAR file as shown in the screen shot below:

new-har-test

Click Start Test and you are good to go!

Keep in mind that HAR files with full response bodies can be quite large. I’ve seen files that are several megabytes in size. Optimization has an uploaded file limit of 30 MB. Luckily, you can also ZIP the HAR file, which will drastically reduce its size. Optimization will happily accept a ZIP file that contains a HAR.

Avoiding problems with browser HARs

There are a few things you need to be aware of when trying to save HAR files with full response bodies and audit them with Optimization.

First, different browser types and versions can do a pretty crappy job creating properly valid HAR files, as Eric Lawerence outlines in The Sad State of HAR. Second, browser plugins or extensions can interface with the browser ability to make network requests, and thus lead to empty, malformed, or incomplete HARs. This is especially true for ad blockers. I recommend launching your browser in Private Browsing mode, which starts you with a empty cache, no state, and by default no plugins or extensions running. In short, there are a lot of ways browsers can create a garbage HAR that may or may not be able to control.

Sadly, that’s not all. The Save HAR with Content feature of these browsers is often buggy and it can be fairly easy to cause it to hang. If the page is still loading, or if there are still outstanding network requests when you try to export the HAR, often the browser network panel will hang, and never finish the export. You’ll just see a progress bar like below, that never actually finishes.

har-progress

This is especially true on media sites, or sites with a lot of third party content, or really anything that continues to make requests after the page is loaded. Make sure there are no progress spinners or anything going on before you try to export the HAR. I’ve found a handful of sites which simply never seem to be able to finish a HAR export. The browser just hangs on the export with a progress bar that never completes.

Making matters worse, browsers like Chrome will create the HAR file on disk before the export is complete. This means that if the export hangs for one of the reasons discussed above, you’ll still see a HAR file on disk and it will have some JSON content in it. However it won’t be complete. Trying to audit this to Optimization won’t work since the JSON would be invalid.

Tips for recording HARs in Browsers

Here is a list that will help you record HARs with full response bodies:

  1. Make sure your desktop browser is up-to-date.
  2. Disable all plugins or extensions, especially ad blocking extensions.
  3. Make sure Disable Cache is checked.
  4. If you want to capture a multi-page process, make sure Preserve Log is checked.
  5. Ensure that the page is fully loaded and there are no pending requests. Clicking the stop button can help.
  6. Make sure you select the export option that includes full response bodies.
  7. Make sure the export progress bar has finished before attempting to upload the HAR file
  8. If the HAR file is too large, put it in a ZIP file and upload the ZIP file to Optimization.

Conclusions

With our new ability to audit HARs with full response bodies, we’ve decoupled the collection of traffic from the analysis of traffic for performance problems. Instead of having to configure Optimization to access a system to retrieve its content, you use your own browser to collect the content and save it in a HAR file. Optimization is just analyzing the content you collected and does not need to have direct access to the website you want to audit. If your browser can see it, Optimization can audit it!

Suggested Blog Posts

The Perception Gap for Poor Web Performance

E-commerce revenue continues to grow,as consumers turn away from shopping in brick-and-mortar stores to shopping online.  However, many businesses are not prepared for this growth because they do not fully understand the market and how to invest in...

Read More

Using Browser Trends to Maintain a Better Site

Because of the multifarious nature of web clients today, it’s important to consider the usage statistics when designing, implementing, and managing your site. However, misconceptions often arise when determining what browsers to design...

Read More

Finding Causes of Intermittent Errors from Google Webmaster Tools

Google Webmaster Tools is a web service that allows webmasters to view the status of their sites as seen by Google and the Googlebot crawlers. In addition to indexing your site structure and content, the Googlebot crawlers also record data on perform...

Read More

Optimization Options not Always Optimal

Web designers and developers are always looking for ways to speed up their page load times. Yahoo has an excellent article written on the best practices for speeding up your page. How do you know if implementing one of their suggested practices will...

Read More