Play

• Time to Load
• Time until usable
• Jitter
• Responsiveness
• Smoothness

Download

# of resources: images, fonts, HTML, scripts, and CSS loaded

Parse

File size of above resources

Execute

Parsing & Painting

Perceived Performance

Users perception of the speed of the load and reaction time.

• HTTP request
• DNS lookup
• TCP Connect
• HTTP request sent
• Server Magic
• Server Sends response
• Browser receives/parses response
• Resources fetched from Cache
• Parse & Execute Scripts
• Render Site

• Each request: go back to DNS lookup or HTTP Request
• Stylesheets are blocking
• Scripts are blocking.

Load Times: 3,729ms v. 3,768ms Play

Visually Complete: 16s v. 8.7s

What is good User Experience?

• How does visual jitter impact perceived performance?
• Are sites free of visual jitter like modals and overlays viewed as more performant?
• Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?
• Does a long DOM Content Loaded impact perceived performance?
• Can metrics be improved?

User Experience > Developer Experience

Metric on above-fold visual Quality of Experience

• Created by Patrick Meenan (Google)
• Used on WebPage Test

Aggregate function on quickness of above-the-fold visual completion:

• 
• 4,462
• 5,902

• Frame-by-frame VC progress is computed from pixel-histogram comparisons

  

• Frame-by-frame VC progress is computed from pixel-histogram comparisons

  

• Pixel-wise similarity (mean histogram difference a.k.a. MHD) doesn’t capture visual perception!
  • Perception of Shape / Color / Object similarity

• Update: Frame-by-frame VC progress computation using SSIM

  

Frame-by-frame VC progress computation using SSIM

• SI and PSI: linearly correlated
• Visual jitter / layout thrashing? PSI > SI
  • PSI appears higher when visual jitter exists (Pop-up ads / large lay-out changes / etc.)
• SSIM based visual progress measurements match human perception more closely than MHD
• SSIM / MHD swap doesn’t affect websites without visual jitter

Speed Index

• Primarily focused on progress of above-fold loading
• Does not account for layout stability

Perceptual Speed Index

• A perceptually oriented metric to measure above-fold visual QoE
• Designed to account for visual jitter (layout stability)
• Complementary to SI

• PSI and SI open sourced on GitHub: github.com/WPO-Foundation/visualmetrics
• For more details and explanations: http://www.parvez-ahammad.org/blog/perceptual-speed-index-psi-for-measuring-above-fold-visual-performance-of-webpages or http://bit.ly/2ldyo9n
• PSI in action: GoogleChrome Lighthouse
• SI in action: WebPageTest.org

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

• 5,444 sessions, of which 51% were complete and valid
• 77,482 votes, of which 75% were valid

Perception of speed and UX strongly impacted by popups / overlays

Hypothesis 3: User will not wait until “Visual Complete” to make their choice (despite the explicit instruction to wait until video turns grey)

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

• Does presence of visual jitter / interstitials interfere with metric performance?
  • Can metrics be improved?
• Will there be different trends for sites that are free of visual jitter like modals and overlays?
• Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?

Hypothesis 1: Visual metrics will perform better than non-visual/network metrics

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

Hypothesis 2: No single metric can explain human choices with 90%+ accuracy

• There appears to be no one unicorn metric but, is there a combination synthetic metric (joint ML model) that will do a better job?

• There appears to be no one unicorn metric but, is there a combination synthetic metric (joint ML model) that will do a better job?
• People only looked two videos and made the call. Is there some additional information that we can extract from videos that will improve our models?

• Challenge: speedperception.meteorapp.com/
• Findings & Repo: github.com/pahammad/SpeedPerception
• Team:
  • Qingzhu (Clark) Gao: Data Scientist @ Funemployed
  • Parvez Ahammad: Head of Data Science & Machine Learning @ Instart Logic
  • Prasenjit Dey: Software Engineer @ Funemployed

Phase-1 crowd sourced 07/28/2016 - 09/30/2016.

• How do visual jitter & interstitials impact perceived performance?
  • Do they interfere with metric performance?
  • Can metrics be improved?
  • Are sites free of visual jitter like modals and overlays viewed as more performant?
  • Is it possible to automatically predict the presence of jitter to help choose a better set of metrics?
• Does a long DOM Content Loaded impact perceived performance?

User Experience > Developer Experience

Lighthouse