XmlListModels in Qt 6

I had a look at a small XmlListModel based project of mine and started migrating the code from Qt 5.12 to Qt 6.2. The code ports pretty cleanly, but there are some caveats to be aware of.

As I’m lazy, I started by changing the imports from 2.12 to 6.2 and tried running the code. The first changes I had to make was to change the import from QtQuick.XmlListModel to QtQml.XmlListModel. I also learned that the import statement no longer requires a specific version to be specified – I’m not sure if I’m a fan of that quite yet.

The second change was that XmlRole has been renamed to XmlListModelRole, and that it no longer has a query property, but an elementName and attributeName property. I guess that saves Qt from having to implement support for XPath queries, and in my use-case (and most others), this should still be enough.

The last change I had to made was to silence a warning. It is no longer encouraged to connect objects directly to signals in QML. In my case, it was animations triggered by the onAdd and onRemove signals in a model. The trick is to declare the animation (in my case, a pair of SequentialAnimation instances, separately. Provide an id for them, and then call start on that id in the signal handler.

All in all, a quite pleasant migration experience with only superficial API changes to handle. All logic could be used as is. Nice!

Looking at Vue

All applications are more or less connected today. The time of files on a disk, or moving them with a USB stick (or floppy) are over. Even file based programs are often synced using Nextcloud, dropbox, google drive, etc.

At Eperoto I’m busy building a backend for a React frontend, but there I stay in my comfort zone at the backend. It is Python, databases and files, just as I know and like things to be. I also have my normal toolbox for debugging and know how to execute a rich set of unit and integration tests to ensure that things stay sane over time.

However, I have another side project. Finally I’ve reached a point where have to do take a dip in the sea of web frontend. I don’t mean messing about with the odd Javascript snippet or fighting the windmil^Wcss.

The choices I was looking at where React, Angular and Vue. I did not do a deep analysis when picking a framework to try. Instead my reasoning was a long these lines:

  • React, I can pick this up from Manos at Eperoto.
  • Angular, requires npm and a setting up an environment to get started.
  • Vue, just needs a script to be included into your web page.

So, me being lazy, I choose to view Vue, which is pronounced view. It seems that I like a bit of pronunciation ambiguity in my frameworks.

Starting Slow

The reason for choosing Vue was the ability to run right in the browser. The strength of today’s browsers is the easy of deployment, but a hidden strength takes me back to the 1980s and 90’s. Everyone has a development environment sitting right in front of them. However, instead of being used to load programs like with the C64, you now I have to look for the development tools to bring up the console.

https://ih0.redbubble.net/image.537660612.0177/pp,550x550.u10.jpg

I’m building a tool that is currently in closed alpha stage, so I cannot tell you too much about it. What I can show is one of my first Vue components. It is a text element, that when clicked turns into a text edit. I made two of these, one for editing a line, and one for editing a paragraph. It also sports a placeholder text if it is empty.

Vue.component('editable-text', {
     props: ['value', 'placeholder'],
     data: function() { return { editing: false, } },
     template: '{{value}}' + 
               '{{placeholder}}' +
               '',
     methods: {
         on_start_editing: function(e) {
             this.editing = true;
         },
         on_end_editing: function() {
             this.editing = false;
         },
     },
     directives: {
         focus: {
             inserted(el) {
                 el.focus();
             }
         }
     }
 });

The code above makes it possible to use an editable-text tag. In order to be able to use the new tag, a Vue app needs to be created and mounted into the DOM of the web page. That means making it take over a div somewhere in the page, and let Vue manage its contents.

This is partially done in the Javascript file:

var app = Vue({
     el: '#app',
     data: [ text: '' ]
});

And the other half goes into the html:

<div id="app">
<editable-text v-model="text" placeholder="enter text"></editable-text>
</div>

<script src="js/vue-test.js"></script>

Finally I sprinkled some css on top of this to make it look sane. For instance, defining a placeholder class.

.placeholder {
     color: #999999;
}

Now we have something that works! I can include a development version of Vue into my page, which provides helpful feedback and point me in the right direction when I make mistakes. And then I can point this to a minified production build of Vue once I’m done.

Bring on the Django

For my backend needs I choose to go with Django. It might because I know Python, it might because of the extensive docs, the great intro tutorial (no damn video – text that I can read back and forth in my own order and pace), or maybe it is just because I’m old.

I like Django. Django plays nice. It is my choice of backend framework. End of discussion.

So, how do we make Vue and Django play nice?

First up, the index.html needs to be served from Django, and the CSRF cookie needs to be served along with it:

from django.views.decorators.csrf import ensure_csrf_cookie
@ensure_csrf_cookie
def my_view(request):
     context = { … }
     return render(request, 'app/index.html', context=context)

Then the index.html needs to be adapted, e.g. use {% static 'app/index.html' %} instead of direct URLs and such. Then I place the js files in the app/static directory tree and run migrate.py collectstatic to get it all in the correct location for nginx to serve it on the production server.

There is, of course, more to the app. I provide a REST-ish API passing JSON around via a set of views served under /api/…. I could probably have used the Django REST Framework here, but I tend to have to adapt things as the API isn’t really proper REST. I just picked out the convenient bits from it, so I roll my own for now. To make these requests, I use Axios, which plays nicely with the Django CSRF cookie.

What is the Point?

When using Vue like this, the question is really what the point is. For smaller stuff I’ve rolled this in plain Javascript. Including the AJAX (AJAJ?) calls and updating the DOM. What Vue helps me with is that I can share a single JSON state between the server and client, and Vue reactively renders it as needed on the client side.

Still, writing Vue as shown above kind of sucks. The template part of a component is html wrapped into a Javascript string. This means loads of fun when handing various string enclosing chars – ” and ‘, I’m looking at you two! Also, just stuffing everything in to a Javascript map makes everything a bit clumsy. It is really had to misspell something and it just stops working.

So, it was time to take it to the next level. Vue can handle single file components. This sounds a lot like classes to an old C++ guy like me, so bring it on!

The Computer Says No

Vue might know what a *.vue file is, but Firefox sure does not. It does not know what an include is either. It has lots of interesting errors about it, so it knows about a similar concept, but not what I needed it to know. Instead, I need a compiler and linker, or what the webistas call a bundler.

When mucking about with npm to get React or Vue things to work, I always ended up with webpack. But apparently web pack is old, boring, and slow (a bit like me). Having asked a friend, I learned that all the new kids use Parcel – a project that has an icon next to every heading on the whole site. Amazing…

What Parcel (and webpack or any other bundler) does is that it transform assets into Javascript, HTML and CSS, that the browser understands. It also combines the assets into fewer files, e.g. one js-file instead of a pile of them, thus reducing the number of files the browser has to request, thus making things a bit quicker.

If this sounds a lot like what a compiler (or transpiler) and linker would do – it is exactly what it is.

So, with a bundler like Parcel, I can build my *.vue files into Javascript, which I then can run in my Browser.

What is the Point – part 2

So, what does a Vue file look like?

<template>
    <span v-if="(!editing) && (!!value)" @click="on_start_editing()" >{{value}}</span>

    <span v-else-if="(!editing) && (!(!!value))" @click="on_start_editing()" class="placeholder">{{placeholder}}</span>

    <span v-else-if="editing">
        <input v-focus="" :value="value" @input="$emit('input', $event.target.value)" @keydown.enter="on_end_editing()" @blur="on_end_editing()" type="text" class="form-control">
    </span>',
</template>

<script>
export default {
    name: "editable-text",
    components: {},
    props: ['value', 'placeholder'],
    data: function() { return { editing: false, } },
    methods: {
        on_start_editing: function(e) {
            this.editing = true;
        },
        on_end_editing: function() {
            this.editing = false;
        },
    },
    directives: {
        focus: {
            inserted(el) {
                el.focus();
            }
        }
    }
}
</script>

Apparently you can stuff some css in there as well. But still, the whole script part is just the same mess as before. Why, when making a transpiler, didn’t anyone come up with a sane syntax for properties, data, methods and directives?

Django and the Modern Frontend

Ignoring my annoyances with the the single file Vue components, it is still an improvement, so let’s continue.

I would like to build my frontend and backend in one go. I’d like a single git hash and know that the frontend and backend there fits together. At the same time, I don’t want to deploy my entire development toolchain to the production server. What I want is:

  • The Django stuff running an an uwsgi application.
  • All static files served directly from nginx.
  • All the frontend stuff to be a set of static files.

Enter: django-compressor-parceljs. Using this Django extension I can have parceljs do it’s work on the fly, in the Django app – and it allows me to build for production as well. More on this later.

However, going for parceljs took me one step beyond what I could do with simply including scripts into my web pages, I needed to install npm. I have a lot to say about this tool. To some extent it feels like it is trying to solve a problem with tooling more complex than the system, but let’s get to some specifics:

  • parceljs (and lots of command line tools) wants to get installed globally. This will collide head on with the distro, so it is a no-go for me.
  • When installing locally (in your dev tree) with npm, the executables end up in node_modules/.bin. A hidden directory. Thanks. Still, fixable by updating PATH.
  • The number of dependencies pulled in for a fairly small project is quite astonishing. The number of dependencies is huge. My package-lock.json clocks in at 6243 lines… That is for Vue, Axios and Parcel.
  • Many dependencies likes to not install their dependencies, but instead to tell me to do so in the form of error messages. I’m not sure I get the point of this – just install it, or have a recommends option or something.

Having setup django-compressor-parceljs and updated my PATH to include node_modules/.bin, it actually works. However, since the vue-files are compressed, the development experience is not perfect. I’d much rather be able to debug the vue-files directly via the Django server. If anyone knows how to, please tell :-)

I also created a small script that sets the COMPRESS_OFFLINE to true and then compil^Wcompresses the frontend on my development machine, meaning that I don’t have to install npm and all the dependencies of vue, parceljs, and axios on the server.

Conclusions

I’m still learning. I bet there is a lot that can be improved in this workflow. I also need to work on my CI/CD to do the offline compression automatically, and such. Still, I have some reflections to make from the viewpoint of a C++, Qt, QML, Python perspective:

  • All my web frontend friends appreciate the separation of behaviour from style from state. This can be done in QML, but requires discipline.
  • To me, the vue files are only half-way there. The script part of the file would benefit from a domain specific language such as QML, instead of putting everyting into a dictionary based on conventions.
  • To a large extent it feels like the web frontend world reinvents terminology – I really hope that there are some people carrying over experiences (in both directions). For instance, a bundler is just a transpiler + linker. Why not reusing common Makefile systems instead of re-inventing the wheel?
  • Why can’t I do nice inheritance and templating within Vue components? I.e. why do I have to duplicate all code for a line edit version of a clickable text when I have the text area version already? What am I missing?

Is anyone else doing this? How does your setup look and what does your working process look like?

Also, Happy Holidays and God Jul!

QmlBook: Felgo Service Integration

Felgo has kindly sponsored the QmlBook, which has resulted in a new chapter. The topic this time around is the Felgo Qt extensions for
integrating various services that are commonly used by app developers, the Felgo cloud builds, as well as their live reloading technology.

When building modern apps there are many things that you might want to integrate – in-app purchases, ads, analytics, user accounts, user settings, real-time sharing of data between devices. Felgo provides integrations of common solutions for this which let’s you focus on developing your app. In the Felgo Plugins chapter, we look at some of them.

Another hassle when developing apps is that you need a Mac to build for Apple devices – unless you use Felgo cloud builds. Felgo cloud builds is a CI/CD solution for building and deploying Qt apps directly to an app store.

In addition to this, the chapter contains a deep dive into the Felgo live reloading solution. We had a quick look at using this in the first Felgo chapter. In this chapter, we look at how you can integrate it into your own executables, as well as how you can use it to develop on multiple devices simultaneously.

Intense weeks

End of October turns out to be one of the highs when it comes to workload this year. Everything happens at once – there are two public events that I’d like to tell you about.

The first one is running lights. This is an annual running competition organized by AIF Friidrott, the sports club my kids are active in. This year, this means organized by me and postponed due to COVID-19, but the virtual races started this weekend and the arena race will take place on the 24th.

If anyone of you are in the Alingsås area and enjoy I highly recommend you to join. The weather looks nice, and we will light up the arena with live fire, so it will be a great evening.

The second one is the foss-north 2020 take II event. This spring, we decided to try to organize a physical foss-north event this fall, as obviously the pandemic must be over by November. This seems to not be the case. :-)

Instead we are running a single day event on November 1 with six handpicked speakers. The event is virtual and free for all.

I would like to tell you about the speakers one by one, because I’m very excited about each and everyone of them.

Andrew 'bunnie' Huang

In the morning we welcome Bunnie Huang who will talk about the precursor project. Precursor is an open hardware platform for secure, mobile communications and computations. The focus is on security aiming to create a trustable platform.

Simon Ser

Next up is Simon Ser. He will talk about how to get pixels onto the screen in a modern Linux stack. This means a deep dive in the Kernel Mode Setting (KMS) interface. How it exposes hardware blocks and how to use it to get images shown on the screen.

Ramón Soto Mathiesen

The morning session then ends with Ramón Soto Mathiesen taking us into the land of Domain Driven Design (DDD) using Algebraic Data Types (ADT). Ramón has a background in functional programming languages and brings this knowledge into the world of multi-paradigm languages such as C#, Rust, and Swift.

Carol Chen

The afternoon session starts with Carol Chen from Red Hat Ansible. She works as a community manager for Ansible. She will be talking about how they move have moved from collections to contributions to conferences.

Lars Brinkhoff

We then continue with Lars Brinkhoff who will talk about the Incompatible Timesharing System (ITS). Lars works with restoring ITS and recreating the history from these early days of computing. ITS is of particular interest at foss-north at is the platform where tools such as Lisp, Logo, Scheme, Emacs and Zork where developed. This is where the foundations for the free software movement where born – quite literally.

Tor-logo by Stanchenko on DeviantArt

The day then ends with Alexander and Georg who will talk about Tor, the anonymity network. They will discuss why diversity is essential for reaching security and anonymity.

So, the next days will be crazy hectic, but it is all for something good. First a cosy evening of running on an arena lit by live fire, and then a day of talks about various FOSS projects.

I hope to see you there!

Felgo in the QML Book

Over the past year I’ve been bumping into the Felgo crew at various Qt events. They take Qt to the next level. It all started as a game development framework for Qt, but has turned into a powerful app development framework taking a lot of the rough corners of Qt, and extending the tooling with a powerful live reloader at the same time.

https://qmlbook.github.io/_images/felgo-logo.png

To showcase how the Felgo SDK fits into the bigger picture, and how much it actually helps creating apps, we agreed to extend the QML Book with a Felgo chapter. The chapter is built around a messaging app and showcases the whole UI creation from initial skeleton to working app.

https://qmlbook.github.io/_images/messaging-stage-2.png

We also cover a bunch of other things such as how to get started, the QML Live reloader, as well as some more advanced topics such as native dialogs, simplified networking and the JsonListModel.

Big thanks goes out to Felgo for supporting this work and helping to make QML Book better.

Photoframe Hack

Sometimes you just want to get something done. Something for yourself.

You do not intend it to be reused, or even pretty.

You build a tool.

My tool was a photoframe with some basic overlays. I wanted the family calendar, some weather information (current temperature + forecast), time, and the next bus heading for the train station.

To make this acceptable in a home environment, I built it as a photoframe. You can find the sources in the hassframe-ui repository on my github.

A hidden feature is that if you tap the screen, a home automation control panel slides up. That way you can control all the lights, as well as heat in the garage and an AC in the bedroom. Very convenient.

All this is built using QML. Three somewhat useful models are available:

  • IcalModel, taking a URL and parsing whatever it gets back as ICAL data. It is a very naive parser and does not care about things such as time zones and other details.
  • YrWeatherModel, uses yr.no‘s public APIs to pull out a weather forecast for a given location.
  • ButStopModel, uses the APIs from resrobot to look for departures to the train station from two bus stops close to my home and then merge the results into a model.

I also have a bunch of REST calls to my local home assistant server. Most of these reside in the HassButton class, but I also get the current temperature from there. These are hardcoded for my local network, so needs refactoring to be used outside of my LAN.

All of these interfaces require API keys of one kind or another – be it a proper key, or a secret URL. These are pulled from environment variables in main.cpp and then exposed to QML. That way, you can reuse the components without having to share your secrets.

All in all the code is quite hacky. Especially main.qml. I refactor out parts from there now and then, but the photoframe works, so its not anything that I prioritize.

Currently it runs on a Raspberry Pi on top of Raspbian. I want to build an optimized Yocto image making it less hacky and more pre-packaged. Perhaps there will be a rainy day this summer and I’ll get around to it. Burkhard has prepared the instructions needed over at embedded use.

The Cost of no Architecture

Like many others, I enjoy various reverse engineering and tear-down stories. Personally, I mean things like iFixit tear-downs and Ken Shirriff’s blog, so I started following this tweet thread by foone.

This continues with another tweet sequence about getting software running on the remote control. Having enjoyed these tweets, I started thinking.

The Harmony remotes are quite expensive in my mind. I can’t find any exact numbers for the number of sold devices, but I found this 2018 Q4 earnings report. Looking at the net sales, I guess the remotes are either “Tablets & Other Accessories” or “Smart Home”. They represent sales net sales of ~107 and ~89 MUSD over 12 months. Let’s pick the lower number and just look at magnitudes. The Harmony 900 seems to have retailed for ~350 USD back when it was new. So, if all the Smart Home stuff was harmonies, we’re looking at 250k units over a year. So I’m guessing the magnitude is around 10k – 100k units annually – but the Harmony 900 is from 2013, so I assume that it sold closer to the lower number, if not below. The market was new and so on.

Then we look at the tweets again. What have we got? Let’s put aside security issues, unencrypted communications, and other clear mistakes and just look at how the device is built.

Flash to drive the UI, double web servers on-board, Lua, QNX and what not. A 233 MHz CPU and ~64MB of FLASH – for a remote control. From an engineering perspective, this sounds like a fun system to work on – from an architecture perspective, it looks like a ball of mud.

Back in 2013, QNX might have been a good choice compared to Linux. Today, with Yocto and similar tools for developing embedded Linux systems, it feels like an odd choice to add a license cost to such a device. But no biggie. Back in the day this was not an unreasonable choice (and still isn’t for certain applications).

The Flash stuff. There were alternatives back in 2013, but sure, there were plenty of developers at hand and things like Qt QML was still probably a bit clunky (I can’t recall the state of it back then – it required OpenGL ES, which I guess was a big ask back then).

But the mix of techniques and tools. The on-board web servers. The complexity of a small system and the costs it brings to maintenance and testability. If this is the foundation for Harmony remotes and a platform that has been used for the better past of the past decade, I wonder if the added engineering costs for architecture the platform to be more optimized early on would not have paid off in lower maintenance costs, as well as lower hardware costs.

I know how it is when you’re in a project. The deadline is there in big writing on one of the walls. You can get something working by stringing what you have together with duktape and glue. The question I’m asking myself is more along the lines of how do we run embedded systems engineering projects? Where did we go wrong? Why don’t we prioritize the thinking and refactoring over the just-get-this-thing-out-of-the-door?

The answer is time to market and such, but over a decade of building on a ball of mud, the economical numbers start adding up in favour for the better engineered product. For continuous improvement. For spending time thinking about how to improve the system as a whole.

What a License Track!

The foss-north 2020 videos are rolling out. This year we’re doing a small experiment, so everything is available at once over at conf.tube, while we roll the videos out gradually at YouTube in an attempt to feed the algorithm (like and subscribe!).

This year we had a great set of licensing related talks, and I’d like to discuss them all in this post.

Monday morning started with Frank Karlitschek and his talk Why the GPL is great for business. This a great overview of how you can build an free and open source business – pros and cons and pitfalls to avoid.

Next up is Gabriel Ku Wei Bin from FSFE who talked about REUSE. The REUSE project is about helping creators choose and apply free and open source licenses.

This is followed by Pavel Kopylov and his talk Hacking the legal code of an open source license. This talk is about understanding how licenses works and how to use them.

This is followed by Jason Hammond from Whitesource talking about their compliance tooling and why compliance is important.

The final talk in this track is by Adriaan de Groot talking about the KDE Free Qt Foundation. This is an interesting aspect, as it is about protecting the customers by offering a more liberal license at a given point of time.

Historically we’ve always split talks on a specific topic during the conference to ensure that people move about in the hallways and that most visitors get to see something unexpected. Since we record everything, we can now do both – clustering by topic and a linear playlist.

More foss stuff

It is busy days at the moment – but in a positive way.

First of all – a huge thanks to everyone who submitted to the Call for Papers for foss-north 2020. We have over 70 hours (!!!) of contents to squeeze into two tracks over two days. As always, it will be hard to pick the speakers to create the very best program.

Other foss-north activities includes starting to populate the community day activities, as well as getting a whole bunch on sponsors onboard. An extra big thanks to Luxoft and Red Hat Ansible for helping us by picking up the Gold Sponsorship packages. Ansible are even running their European Contributor Summit as a part of the foss-north Community Day together with events by KDE, Gnome, FreeBSD, Debian, and a hardware hacking workshop. I’m really looking forward to this – if you want to join in with your own project, workshop, hackaton, etc – just ping me!

The other big foss-north change for this year is that we are finally abandoning Eventbrite for a self-hosted system. Big thanks to Magnus Hagander helping us getting the pgeu-system up and running. At the moment, we offer login via Github and Google OAUTH. We’re looking into setting up a self-hosted OAUTH service as well, to let you log in locally, but that will not happen for the 2020 event due to time reasons.

Closer in time is the next local foss-gbg meetup. We are running an event around React together with our good friends at Edument. We already have 50+ registered attendees, so it will be fun!

In other news – I’ve also released Ordmonster – if anyone has kids who wants to get started reading. This is a complement to the Mattemonster app for basic maths launched earlier. Both are made with Godot, a tool that I enjoy more and more.