I recently announced gerrymander, a python client tool and API for extracting information from Gerrit, intended to replace my use of various other ad-hoc tools that people have developed. One of the things I observed with using the previous qgerrit command line tool was that there were a couple of different sets of command line arguments that I used all the time. This was tedious so I ended up creating a number of shell wrapper scripts / aliases to invoke qgerrit with different sets of arguments. With gerrymander though, I wanted to do things a little better by defining this stuff as data rather than code. ie using configuration files, instead of wrapper scripts/aliases.
Taking the ‘changes’ command as an example, it is a generic command for displaying lists of changes in gerrit for a given query. The default behaviour is to output 7 columns: status, url, owner, subject, creation date, last updated date and approvals.
$ gerrymander changes -p openstack/nova-specs
Changes
-------
+-----------+------------------------------------+-----------------------------+-----------------------------------+----------+----------+-------------------------------+
| Status | URL | Owner | Subject | Created | Updated | Approvals |
+-----------+------------------------------------+-----------------------------+-----------------------------------+----------+----------+-------------------------------+
| MERGED | https://review.openstack.org/80785 | russellb | Add README and base directory ... | 68 days | 66 days | w=1 v=2 s=1 c=2,2 |
| NEW | https://review.openstack.org/80865 | mikalstill | Proposed blueprint for libvirt... | 67 days | 1 days | v=-1 c=1,1,-1,-1,-1,-1 |
| ABANDONED | https://review.openstack.org/80866 | mikalstill | Proposed blueprint for live mi... | 67 days | 55 days | v=1 c=-1,-1 |
| MERGED | https://review.openstack.org/81381 | jogo | Add Apache 2 License | 65 days | 63 days | w=1 v=2 s=1 c=2,1,2 |
...snip....
Personally I’m not too interested in the last updated date, status or owner most of the time, and want things sorted by last updated date. I want to give lots of space for the subject, have approvals to be displayed in colour and only display changes that are open and in the master branch. This can be done using command line arguments but it gets very tedious:
$ gerrymander changes -p openstack/nova-specs \
--field url --field subject:80 --field createdOn --field approvals \
--color --branch master --status open --sort createdOn
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/80865 | Proposed blueprint for libvirt serial console work in juno. | 67 days | v=-1 c=1,1,-1,-1,-1,-1 |
| https://review.openstack.org/81828 | Scheduler: Adds per-aggregate filters | 63 days | v=1 c=1,1,-1,1 |
| https://review.openstack.org/82456 | Propose: Normalize Weights (adapt weighers) | 60 days | v=-1 c=-1 |
| https://review.openstack.org/82584 | Proposed blueprint for libvirt Sheepdog instances. | 59 days | v=1 c=1,2,1,1 |
...snip....
This output preference can be added to the $HOME/.gerrymander config file directly, avoiding the need to create a wrapper script
[command-changes]
field=url, subject:80, createdOn, approvals
branch=master
sort=createdOn
color=true
With the result that this is now the default
$ gerrymander changes -p openstack/nova-specs
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/80865 | Proposed blueprint for libvirt serial console work in juno. | 67 days | v=-1 c=1,1,-1,-1,-1,-1 |
| https://review.openstack.org/81828 | Scheduler: Adds per-aggregate filters | 63 days | v=1 c=1,1,-1,1 |
| https://review.openstack.org/82456 | Propose: Normalize Weights (adapt weighers) | 60 days | v=-1 c=-1 |
| https://review.openstack.org/82584 | Proposed blueprint for libvirt Sheepdog instances. | 59 days | v=1 c=1,2,1,1 |
...snip....
Now the obvious limitation with setting defaults in the config file is that you might have many different sets of defaults you care about. For example when looking at nova-specs designs, I’d prefer the sorting to be done by subject and don’t really care about the author. The gerrymander config file copes with this too, by allowing you to setup command aliases:
[commands]
aliases=nova-specs, cinder-specs
[alias-nova-specs]
basecmd=changes
help=Nova design specs
[alias-cinder-specs]
basecmd=changes
help=Cinder design specs
These new commands are now first-class citizens on a par with any other built-in commands as far as gerrymander is concerned, so you can then define default settings for them independently.
[command-nova-specs]
project = openstack/nova-specs
status = open
field = url, subject:80, createdOn, approvals
sort = subject
color = true
[command-cinder-specs]
project = openstack/cinder-specs
status = open
field = url, subject:80, createdOn, approvals
sort = subject
color = true
Trying out the new nova-specs command…
$ gerrymander nova-specs
Changes
-------
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| URL | Subject | Created | Approvals |
+------------------------------------+-------------------------------------------------------------------------------------+----------+--------------------------+
| https://review.openstack.org/95054 | API: Live Resize | 9 hours | v=1 |
| https://review.openstack.org/85726 | API: Metadata Service Callbacks | 45 days | v=1 c=1 |
| https://review.openstack.org/85673 | API: Proxy neutron configuration to guest instance | 24 days | v=1 c=1 |
| https://review.openstack.org/94918 | Add Barbican wrapper specification | 20 hours | v=1 |
| https://review.openstack.org/94370 | Add LVM ephemeral storage encryption specification | 2 days | v=1 c=1,1,-1 |
...snip....
This blog post has just looked at the ‘changes’ command, but any gerrymander command can be customized in this way – every single command line option is mapped into the configuration file in the same way.
Those who saw my original announcement of gerrymander will know that OpenStack uses the Gerrit tool for code review. One of the efforts made by the Nova (Compute) team during the last release cycle was to increase the automated testing coverage for hypervisors supported by Nova. Previously we only tested Libvirt on KVM, but now there is testing for VMWare, HyperV and XenAPI too. This is great for our code quality, but far less great for the usability of the gerrit web interface. Consider a fairly typical change I needed to look at today, the web UI for gerrit shows many comments. When doing code review of new patch versions though, I mostly don’t care about comments from the “robot” accounts doing automated testing – just the pass / fail status. I’m far more interested in comments made by humans, so I can see whether they’ve been addressed by new patch versions, or whether I agree/disagree with the comments. The gerrit web UI though doesn’t provide any way to separate this info – it just shows a huge list with human & robot comments combined. The ever growing number of robots used by OpenStack for testing mean the human comments are drowned out in the noise
This problem is one of the reasons I created the gerrymander tool. The ‘comments’ command has the ability to display all comments on a change, while filtering out those made by known bot accounts. Now there’s a little setup required, by which I mean you must edit $HOME/.gerrymander to list all the bot accounts. For those using OpenStack though, I’ve uploaded a sample config file, which lists all bot accounts, to the wiki. With this config file in effect I can now look at this same change number 87329
# gerrymander comments --color 87329
Change https://review.openstack.org/87329 (I06c2d9930e0f36a0d7057b6a0f5c9c591caac43f)
libvirt: Use os_command_line when kernel_id is set
Patch 1 (c8d1cf4559174777be4b42b68379cf78ccd8f382)
Jay Pipes: (jaypipes) nova/virt/libvirt/driver.py:3203
You can condense lines 3202-3203 to: guest.os_cmdline =
img_props.get('os_command_line')
Jay Pipes: (jaypipes)
Patch Set 1: I would prefer that you didn't merge this (1 inline comment)
Suggestion for consolidation inline, otherwise, looks good.
Vladan Popovic: (vladan)
Patch Set 1: Thanks for the review Jay.
Patch 2 (58323a7271e0f09689b94bc8632204c179934add)
Daniel Berrange: (berrange) /COMMIT_MSG:18
"AMI images" is a pretty obscure/non-obvious term. This is about images with
an explicit boot kernel set, so please say that explicitly and void the term
"AMI"
Daniel Berrange: (berrange)
Patch Set 2: I would prefer that you didn't merge this (1 inline comment)
Jay Pipes: (jaypipes)
Patch Set 2: recheck bug 1307344
Patch 3 (8f3ced280d80eaee69029f4bf2a193bcba749284)
garyk: (garyk) nova/tests/virt/libvirt/test_libvirt.py:1961
please use self.assertIsNone
Mohammed Naser: (mnaser)
Patch Set 3: Looks good to me, but someone else must approve
garyk: (garyk)
Patch Set 3: I would prefer that you didn't merge this (1 inline comment)
code looks good. one minor nit
Patch 4 (5698a6c7c5195bd1d3ecf01af6fb65c19ae8a990)
No comments
Patch 5 (a87773776a0d39aa5591fde6caa65b92f5b17a6d)
Jay Pipes: (jaypipes)
Patch Set 5: Looks good to me, but someone else must approve ++, thx Vladlan!
Mohammed Naser: (mnaser)
Patch Set 5: Looks good to me, but someone else must approve recheck bug
1307344
Daniel Berrange: (berrange)
Patch Set 5: I would prefer that you didn't merge this Jenkins failures are
genuine bugs
Sreeram Yerrapragada: (syerrapragada)
Patch Set 5: recheck-vmware
Patch 6 (a1eb12f0c8281c0b01dde00b8225d742b8e832e3)
garyk: (garyk)
Patch Set 6: Code-Review+1
The obvious limitation here is that the file comments are not shown inline with the code, but overall I think this is still much more useful than the gerrit web UI display of comments. I can still go look at the web UI if I do need to see the context of certain inline comments. So gerrymander comments report complements the web UI nicely.
Following quickly on from my announcement of release 1.0, I’m pleased to announce release 1.1 of gerrymander. Gerrymander provides a python 2 and python 3 compatible API for extracting interesting information from gerrit about changes, along with a command line tool exposing several reports. If you use gerrit for any non-trivially sized project, this tool has the potential to significantly improve your review workflow efficiency. For those working on OpenStack, I’ve created a sample configuration file that includes all the core openstack teams and bot accounts.
Thanks to contributions from a number of interested parties this release has a bunch of small fixes included:
- Fix syntax for passing username to ssh
- Fix syntax error in patch review stats reprot
- Fix misc typos in help / example configs
- Ignore unexpected change type
- Expand ~ in config file paths
- Avoid duplicate names in AUTHORS file
You can get it from pypi
# pip install gerrymander
Or straight from GitHub
# git clone git://github.com/berrange/gerrymander.git
This blog post is to announce the first release of a new project, gerrymander, which I’ve mentioned on IRC in passing a few times. The 32,000 ft summary is that it provides a set of python (2 & 3 compatible) APIs and command line tool for extracting and presenting information from gerrit. You can get it from pypi
# pip install gerrymander
Or straight from GitHub
# git clone git://github.com/berrange/gerrymander.git
If you’re the impatient type, then go to the README file which provides a quick start guide to using the tool.
For details on why/how I built gerrymander, read onwards….
Background motivation
The OpenStack project uses Gerrit for review of pretty much all contributions to the project. Not wishing to get into a discussion of the merits of using a Gerrit based workflow, I’ll just say that with large volume of changes going through some sub-projects, such as Nova, the Gerrit web interface really starts to show its limitations. As a result a number of OpenStack contributors have developed cli tools for extracting information from Gerrit and presenting it in more practical formats.
- gerrit_view – created by Josh Harlow, it provides a general purpose query tool and a interactive TUI for live monitoring of changes.
- reviewtodo – created by Russell Bryant, it generates reports which attempt to prioritize changes such that the most “important” ones are presented at the top of the todo list.
- reviewstats – created by Russell Bryant, it generates reports which summarize the reviewing activity of all contributors across the project, and reports which attempt to identify how efficiently reviews are being handled.
I’ve previously contributed to the gerrit_view project and in the second half of the IceHouse dev cycle, I turned off gerrit email alerts and stopped using the gerrit web UI index pages for identifying changes needing review. Instead I exclusively use the qgerrit command line tool to identify changes that affect the libvirt driver which need attention from myself. This had a significant positive impact on my productivity when using gerrit, so I started looking at other gerrit client tools and thinking about what further reports or information I might wish to get from gerrit. It became apparent that the tools people are writing have significant overlap / duplication of code for dealing with basic interaction with gerrit. There are features in some tools (eg caching of gerrit queries in reviewstats) which would be useful to the other tools, but since these are all designed as singe-purpose standalone tools there’s not really much scope for sharing functionality.
Introducing the “gerrymander” project
Clearly what was needed was a new “standard” for building gerrit command line tools, so enter “gerrymander“. The gerrymander project is not simply another command line tool, rather it is intended to provide a collection of python modules / APIs to facilitate the creation of arbitrary gerrit command line reports/tools. With that goal in mind the gerrymander package provides a set of modules, for both Python 2 and Python 3:
- gerrymander.client – module providing a class for connecting to the gerrit server over SSH, running the ‘gerrit’ tool and passing the results to a callback for processing. As well as the standard “live” client, there is a caching client which stores the results from ‘gerrit’ in local files. This means that expensive queries (eg querying the entire history of all changes ever) won’t inflict repeated denial of service attacks on the server.
- gerrymander.model – module providing a set of classes that represent the JSON schemas returned by the ‘gerrit’ tool as Python objects. This means you’re not simply blindly accessing untyped dictionary fields. Many of the classes have helper APIs against them to allow their information to be accessed in interesting ways.
- gerrymander.format – module providing a few helper APIs for formatting data to present to the user. For example, a way to produce coloured text for ANSI capable terminals, or to format time deltas / dates in more user friendly ways (ie “4 days ago” instead of “May 5, 2014”).
- gerrymander.operation – module providing a class for each operation supported by the ‘gerrit’ tool. This provides a slightly higher level way to utilize to the gerrymander.client module classes. This takes care of obscure oddities such as the need to re-execute ‘gerrit query’ multiple times, since it refuses to return more than 500 results at a time.
- gerrymander.reports – module providing a set of classes for extracting interesting information from gerrit. Each report class will execute one of more operations against gerrit, post-process the data from the query, and then return an object with the structured results. This is where all the really interesting functionality lives.
- gerrymander.commands – module providing the command line interface to the reports. It takes the report output and formats it as text, xml or json. A configuration file is used to customize default behaviour, such as which fields are visible, defining command lines, project names, usernames of bots, etc.
The actual “gerrymander” command line tool is designed as a multi-call binary – ie it has a number to sub-commands you can execute, each with their own set of options. I won’t repeat what’s already covered in the README file in this blog post, rather just see the help message for the list of reports/commands I’ve written. So far I’ve targeted the functionality provided by the 3 pre-existing projects I mentioned above:
$ gerrymander --help
usage: gerrymander [-h] [-c CONFIG] [-d] [-q]
{watch,todo-noones,todo-anyones,todo-mine,todo-others,patchreviewstats,openreviewstats,changes,comments,changes-nova-specs}
...
Gerrymander client
positional arguments:
{watch,todo-noones,todo-anyones,todo-mine,todo-others,patchreviewstats,openreviewstats,changes,comments,changes-nova-specs}
watch Watch incoming changes
todo-noones List of changes no one has looked at yet
todo-anyones List of changes anyone has looked at
todo-mine List of changes I've looked at before
todo-others List of changes I've not looked at before
patchreviewstats Statistics on patch review approvals
openreviewstats Statistics on open patch reviews
changes Query project changes
comments Display comments on a change
changes-nova-specs Changes in Nova SPECS
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Override config file (default
/home/berrange/.gerrymander)
-d, --debug Display debugging information
-q, --quiet Supress display of warnings
As one example, taking the simplest / most generic ‘changes’ command, lets see all changes that I’ve written but abandoned
$ gerrymander changes --owner berrange --status abandoned
Changes
-------
+-----------+------------------------------------+----------+-----------------------------------+----------+----------+------------------+
| Status | URL | Owner | Subject | Created | Updated | Approvals |
+-----------+------------------------------------+----------+-----------------------------------+----------+----------+------------------+
| ABANDONED | https://review.openstack.org/8528 | berrange | Allow CPU model to be specifie... | 694 days | 688 days | v=-1,1 c=-1 |
| ABANDONED | https://review.openstack.org/9356 | berrange | Fix use of uninitialized varia... | 673 days | 672 days | v=-1 c=1 |
| ABANDONED | https://review.openstack.org/10871 | berrange | Revert "Handle InstanceNotFoun... | 641 days | 641 days | v=-1 |
| ABANDONED | https://review.openstack.org/18869 | berrange | Merge LibvirtOpenVswitchVirtua... | 490 days | 486 days | |
| ABANDONED | https://review.openstack.org/19127 | berrange | Merge all VIF classes into one... | 486 days | 463 days | v=1,1 c=1 |
| ABANDONED | https://review.openstack.org/19214 | berrange | Make it possible to set nova o... | 485 days | 483 days | v=1 c=1,-1 |
| ABANDONED | https://review.openstack.org/29784 | berrange | Make devstack work on Fedora 1... | 353 days | 336 days | v=1 c=-1,1,-1,-1 |
| ABANDONED | https://review.openstack.org/58494 | berrange | Increase min required libvirt ... | 164 days | 79 days | v=-1 c=-1 |
| ABANDONED | https://review.openstack.org/76902 | berrange | Fix quoting of username in pol... | 70 days | 63 days | v=1 c=-1 |
+-----------+------------------------------------+----------+-----------------------------------+----------+----------+------------------+
Some things to note about that are unique in comparison to other gerrit client tools I’ve mentioned above that can do the same kind of query
- The gerrit query is cached for 5 minutes, so if you re-run to change the display options (eg which fields are shown) it won’t hit the gerrit server again, unless you change the actual args to the query.
- The default output mode is formatted text, but you can ask for the data in XML or JSON documents, allowing easier parsing by further downstream tools
- The configuration file lets you set defaults for all of the command line parameters. So you can hide fields you don’t care about, or make fields wider, and more
- The configuration file lets you define command aliases. So if you have a number of different queries you run, you can define new commands (eg ‘my-abandoned-changes’) which record all the query parameters for the ‘changes’ command. This avoids the need to create shell wrapper scripts around the gerrymander command for common queries
- The command and/or report are accessible via the Python API, so if you want direct access to the raw data you can use the API instead of parsing the text/xml/json outputs.
- Optional colourization of fields (eg +1’s / +2’s in green, -1’s / -2’s in red)
If any of this sounds interesting to you, pip install the package and try it out. If you want to contribute patches for more interesting reports, then the code is all up on github at the URL mentioned earlier.
One final important point is that this tool is written such that it has zero knowledge about OpenStack. It is intended to be useful to any project which is using Gerrit for their code review. As such all the projects specific knowledge, such as list of project names, team members, bot accounts, is isolated in the configuration file. So one thing I need to do to ease first time users is to upload a sample configuration file for OpenStack that includes all the different projects / teams / bots OpenStack has.
EDIT: Use this config file with OpenStack
I am pleased to announce a new release 0.6.0 of Entangle is available for download from the usual location:
http://entangle-photo.org/download/
This release has a mixture of bug fixes and new features / changes and new widget styling
- Add a demonstration plugin for setting up a captive photo box display mode
- Switch to use Python3 for plugin engine instead of JavaScript
- Use GTK dark theme
- Require GNOME symbolic icon theme to be installed
- Switch to require lcms2 instead of lcms
- Move application icon into a standard directory mandated to work with the appdata tools
- Make manual focus work with Canon EOS cameras
- Disable flickering progress bar in preview mode with Canon EOS cameras
- Remove use of deprecated GTK methods/classes/constants
- Remove use of gexiv2 method which is not long exported
- Remove use of deprecated libpeas methods
- Add GTK-DOC transfer annotations / docs to all methods
- Avoid loosing camera capabilities on disconnect
- Fix off by one in histogram tables causing memory corruption
- Mark appdata / desktop files for translation
- Fix typos in README file
- Fix inverted tests when checking if range widget changed
- Avoid storm of expose events due to auto-drawer widget
- Avoid never ending circular update of controls causing errors in some camera modes
- Add workaround for crazy D5100 camera serial number
- Add customizable highlight/background for images
- Avoid reference leak of windows preventing proper cleanup
- Remove camera manual/about/driver help windows since it did not contain any info useful to users
- Filter list of cameras in connect dialog to only those which support capture/preview
- Don’t auto connect to cameras which don’t support capture or preview
- Ensure parent window is set on dialogs to prevent them falling behind main window
- Fix crash with latest GTK due to incorrect overriding of GtkApplication startup method
- Update to cope with changed GExiv API version
- Refreshed translations from transifex
Thanks to the great work of the Fedora translation team, Entangle has > 90% translation coverage for Dutch, Polish, Ukrainian, French and Japanese and > 80% coverage for Swedish, Czech, Hungarian, Italian, Spanish, Chinese and German. If you are in a position to help translate Entangle further, consider joining the Fedora translation team.
