FullStack Me

Curiosity driven journal of perfecting a comprehensive and mindful living

Progressive images for the progressive framework: When Flutter meets Cloudinary

13 January 2020

Let’s be honest: getting content instantaneously became today’s normal. Be it a wall of text or a wall of images, users want to see high-quality data on their devices straight away. Take favourite social networks: they rigorously fight for their content to be delivered as quick as possible so that the user would get all the finest of pixel data. With things like 5G coming to our lives or fibre broadbands available at home or in the office, this is no longer an unsolvable challenge: people can scroll through hundreds of Instagram photos on the go. Not to mention, it becomes incredibly annoying to get stuck on something and wait while the next post loads, doesn’t it?

Web of progressive images

In the unending fight for the seamless user experience, some platforms went even further than just serving images up to the size of a user agent’s window width. In fact, well-known Medium & Quora became a sort of a “gold standard” in delivering static graphical content. This blurry transition often takes just a fraction of a second and looks so natural that many started thinking this is how all articles or posts should be received now and on.

Not a surprise that this CSS/JavaScript trick got a wide adoption on other web platforms and even within JS frameworks. For instance, Gatsby, an engine for statically served websites (like the one you read at the moment -- noticed that blurry image on top?), uses a low-level image processing JS library called Sharp. Together with some higher-level Gatsby wrappings, this allows me to set an image right from the local GraphQL response like this:

<StaticQuery
 query={graphql`
   query {
     placeholderImage: file(relativePath: { eq: "gatsby-astronaut.png" }) {
       childImageSharp {
         fluid(maxWidth: 300) {
           ...GatsbyImageSharpFluid
         }
       }
     }
   }
 `}
 render={data => <Img fluid={data.placeholderImage.childImageSharp.fluid} />}
/>

And I will see some crazy stuff in page source, like that:

<div class="article-cover gatsby-image-wrapper" style="position: relative; overflow: hidden;">
    <div style="width: 100%; padding-bottom: 56.25%;"></div>
    <img src="data:image/png;base64,iVBO..I=" alt="" style="position: absolute; top: 0px; left: 0px; width: 100%; height: 100%; object-fit: cover; object-position: center center; opacity: 0;">
    <picture>
        <img sizes="(max-width: 960px) 100vw, 960px" srcset="/static/ae46cfa9b114c6bb91ad0c9a9de03269/50db9/json-1.png 240w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/9d65d/json-1.png 480w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/bc549/json-1.png 960w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/2513a/json-1.png 1440w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/ec873/json-1.png 1920w" src="/static/ae46cfa9b114c6bb91ad0c9a9de03269/bc549/json-1.png" alt="" loading="lazy" style="position: absolute; top: 0px; left: 0px; width: 100%; height: 100%; object-fit: cover; object-position: center center; opacity: 1; transition: none 0s ease 0s;">
    </picture>
    <noscript>
        <picture>
            <img loading="lazy" sizes="(max-width: 960px) 100vw, 960px" srcset="/static/ae46cfa9b114c6bb91ad0c9a9de03269/50db9/json-1.png 240w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/9d65d/json-1.png 480w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/bc549/json-1.png 960w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/2513a/json-1.png 1440w,
/static/ae46cfa9b114c6bb91ad0c9a9de03269/ec873/json-1.png 1920w" src="/static/ae46cfa9b114c6bb91ad0c9a9de03269/bc549/json-1.png" alt="" style="position:absolute;top:0;left:0;opacity:1;width:100%;height:100%;object-fit:cover;object-position:center"/>
        </picture>
    </noscript>
</div>

Of course, it does much more than just getting a small preview and replacing with the full-sized quality image, and can offer plenty of features from rescaling to various image encoding/decoding options. Speaking of which, there is quite a wide range of tasty things to try: from quite mature progressive JPEG to new-ish WebP introduced by Google about a decade ago, and many other, not so well-known. A silver lining that connects them all is the idea of serving a little portion of bytes “progressively”: with smaller thumbnail-like preview image encoded in it, and then gradually supply more and more data, ending up with loading the entire image. Interestingly, the idea also benefits from “dynamic programming”-like concept which supposes reusing bytes of smaller-sized images when rendering the bigger picture. Sorry my weak explanation, but I have a good reference for you: Jason Grigsby explained this idea in detail.

So, it sounds like a no-brainer, just go and use all these “progressive” technologies, and the problem is solved, right? Well, there are two little problems here, namely encoding and decoding:

  1. Firstly, the imaging data to be encoded “progressively” so it’d merely contain the necessary bytes put in the necessary order.
  2. One would also need to decode it properly and render as it comes part by part.

Both problems make it quite a tough task to spread the technology around, especially given both continuously growing connection speed & coverage of fast networks along with a great variety of devices and platforms that surround us.

Welcome to the devices world

This is what mainly triggered me to start playing around this topic and inducted writing this post. After trying out a framework for cross-platform apps development called Flutter, I faced the very problem we talk about here: how to serve images in a way comfortable for a user: mainly, show the preview and then progressively load the rest of bytes. To figure this out, I started this small experiment.

If you think about it for a little while, it becomes quite apparent that the problem of smooth images loading is even worse in the devices world. Let me name just a few reasons:

  • A vast diversity of technologies makes it harder to establish a new standard that would be common across different platforms.
  • More complex decoding techniques could take a bit more CPU time and memory, which are ultimately critical resources for mobile devices.
  • Whilst expecting the content coming rapidly fast, users could be within the area of a poor network signal (to see the difference, try throttling the network to 2G in your browser’s dev tools, for example).
  • Despite being compact devices, some of the modern phones feature quite a high screen resolution, in the end requiring more pixel data to be loaded onto the device.

This is why, for example, WebP remains used mostly by Android-based devices. Or why there is so much excitement around progressive web apps (PWA) nowadays, as developers could stick to the well-known and reliable technology stack. Of course, progressive image loading is a solved problem if you develop for one of these platforms: there are powerful frameworks like Fresco for Android and Concorde for iOS that do exactly what you need: load a preview first and then replace it with the full-scale one, or even support one of the progressive image formats. Although, it is still up to you to prepare necessary encoded images or thumbnail previews and find a way to do caching. Furthermore, there is even a way of solving the problem with Flutter using things above: a solution would roughly look like this, exaggerated (see more at https://flutter.dev/docs/development/platform-integration/platform-channels):

if platform is android:
    // get fancy webp
elif platform is iOS:
    // decode progressive jpeg smoothly
else:
    // do something custom

Not pretty.

Generally, with Flutter framework coming, there was (and probably still is) plenty of scepticism how a single tool could fight this hydra of such divergent technology stacks between iOS, Android, desktops, web and whatnot (see a related issue: https://github.com/flutter/flutter/issues/14966). And I guess this progressive imaging problem is just a little one on the battlefield, yet still important for smooth user experience.

Back to basics

Since we talk Flutter here, maybe it’s a good time to throw in some code and look at the problem from the practical side. A sample app that loads images as user scrolls could look like this:

import 'package:flutter/material.dart';
import 'package:flutter/widgets.dart';

void main() => runApp(MyApp());

@immutable
class MyApp extends StatelessWidget {

 List<String> cards = [
   'https://res.cloudinary.com/demo/image/upload/beach_huts.jpg',
   'https://res.cloudinary.com/demo/image/upload/sample.jpg',
 ];

 @override
 Widget build(BuildContext context) {
   const String title = 'Flutter Demo';
   return MaterialApp(
     title: title,
     theme: ThemeData(
       primarySwatch: Colors.blue,
     ),
     home: Scaffold(
       appBar: AppBar(
         title: const Text(title),
       ),
       body: cardsWidget(),
     ),
   );
 }

 Widget cardsWidget() => ListView.builder(
   itemBuilder: (BuildContext context, int index) {
     int cardIdx = index % cards.length;
     return Card(
       child: Column(
         mainAxisSize: MainAxisSize.min,
         children: <Widget>[
           Image.network(cards[cardIdx]),
           ListTile(
             title: Text('Full-size Image'),
           )
         ],
       ),
     );
   },
 );
}

It fires a new request to get image bytes every time user scrolls up or down: definitely works, but could do better. Let’s hold on for a moment and think once again: the goal we want to achieve here is to load an image in a way that the user won’t suffer much waiting. Written down, the list of specific things we want to achieve can look like this:

  1. Display fast: show at least something so the user won’t stare at the loading indicator
  2. Load exactly what we need: avoid getting the one-fits-all image to save time & traffic
  3. Require a bare minimum of work to prepare served images on the backend side
  4. Be cross-platform: please no custom decoders or platform-specific code

At a glance, the very first step could be just getting a smaller image and then loading the bigger one. Isn’t it exactly what Medium does, for instance? In fact, it’s a state of the art technique known to some of you as "blur-up" (https://css-tricks.com/the-blur-up-technique-for-loading-background-images/):

<div class="kp r">
    <div class="bk kj s t u kk ai bd kl km">
        <img class="s t u kk ai kq kr ck qm" src="https://miro.medium.com/max/30/0*V_l2oXB9sHEU6w7d.jpg?q=20" role="presentation" width="960" height="640">
    </div>
    <img class="pa ql s t u kk ai ge" role="presentation" src="https://miro.medium.com/max/960/0*V_l2oXB9sHEU6w7d.jpg" width="960" height="640">
    <noscript></noscript>
</div>

Two insightful things here: first, there is a preview of a very low-quality JPEG. But if you open one, you’d see a tiny image (30-pixel width) that seems to be stretched to the size of the full one. It weighs just a bit over 1KB and gets loaded instantaneously! Another thing, the request seems to specify the width for the full-size image as well. Come on, let’s try the same!

Well… Hang on for a second: how are we supposed to wait for the request completion? Hopefully, there is a smart solution that involves quite a neat load state handling: flutter_cached_network_image. And as a bonus, we get caching of the same image for free!

Widget preloadWidget() => ListView.builder(
 itemBuilder: (BuildContext context, int index) {
   int cardIdx = index % cards.length;
   return Card(
     child: Column(
       mainAxisSize: MainAxisSize.min,
       children: <Widget>[
         CachedNetworkImage(
           imageUrl: "https://res.cloudinary.com/demo/image/upload/beach_huts.jpg",
           placeholder: (context, url) {
             var width = MediaQuery.of(context).size.width;
             return Container(
                 width: width,
                 height: width * 0.65,
                 foregroundDecoration: BoxDecoration(
                   // A manually created thumbnail of 30px width and Q 80%
                   image: DecorationImage(
                     image: NetworkImage("https://res.cloudinary.com/demo/image/upload/beach_huts_w30_q80.jpg"),
                     fit: BoxFit.fill,
                   ),
                 ),
               );
           },
         ),
         Image.network(cards[cardIdx]),
         ListTile(
           title: Text('Full-size Image With Thumbnail'),
         )
       ],
     ),
   );
 },
);

Wow, it started looking “progressive” and just requires generating one preview image for each of the ones we serve. But now look closer and think again: Medium has a fixed layout of 960px width that is nearly the same among all browsers, plus maybe another one for mobile. Overall, this is a well-known problem for the web, which is often called “responsiveness” and it has a number of quite good, reliable solutions like using responsive breakpoints. In the world of phone apps, we could have a number of different resolutions in theory, but only one at a time: should we prepare an image for each of them?

Add the missing bits: enter Cloudinary

To look for help, let’s appeal to progressive clouds. In fact, we need a very specific thing: serve an image of a custom width, and also a tiny thumbnail, preferably with the lowest quality to compress even more. The one I was using with my personal blog for quite a while is Cloudinary, and luckily it allows doing this in a surprisingly simple way. With HTTP API, one just needs to specify the transformation parameters i.e. width and image quality. Now all pieces of the puzzle come together:

  • We have a preview image that is loaded immediately when the user scrolls to it
  • A full-size image is served with the right width in meanwhile
  • Both require only one copy of an image to be stored in the cloud
class ImageCard {
 static const baseUrl = "https://res.cloudinary.com/demo/image/upload";

 String title;
 String image;

 ImageCard(this.title, this.image);

 Widget toWidget(double width, int seed) => Card(
   child: Column(
     mainAxisSize: MainAxisSize.min,
     children: <Widget>[
       CachedNetworkImage(
         imageUrl: "$baseUrl/w_${width.toInt()}/l_text:Arial_35:$seed/${this.image}",
         placeholder: (context, url) =>
             Container(
               width: width,
               height: width * 0.65,
               foregroundDecoration: BoxDecoration(
                 image: DecorationImage(
                   image: NetworkImage('$baseUrl/w_30/${this.image}'),
                   fit: BoxFit.fill,
                 ),
               ),
             ),
       ),
       ListTile(
         title: Text(this.title),
       )
     ],
   ),
 );
}

As you can see, the code above is fully cross-platform, and there are no specific frameworks involved. Strictly speaking, here we rely only on Flutter, the caching lib and Cloudinary, so for me, it sounds like a fair, minimalistic choice to enable a smooth, consistent and truly cross-platform experience.

Read more

Back to other articles