I can build, but I am certainly not the fastest. Slowly but surely, with help from lmacken and threebean and qalthos, I've been loading up the Commops toolbox.

CommOps Tools in the Box

cardsite

One of the first data visualizations I worked on after being hired, this was one of the fruits of the PyCon 2015 Sprints. Cardsite displays fedmsg activity as a 3 part grid, with the top pane being messages, the middle pane being users, and the bottom pane being packages. It creates the cell if the message is new and hasn't been seen before, or, if it has been seen, updates the message count, and does a fancy animation. It is a proof of concept, and not what I would call "complete," but it has a number of merits:

1. Inspired by the EmojiTracker project, that shows real-time usage of Emoji on Twitter
2. Gulp/Bower to install JavaScript/CSS dependencies in a programmatic way
3. Deployed to GitHub via gh-pages branch: http://decause.github.io/cardsite
4. Uses Websockets to provide real-time fedmsg updates to the page
5. Uses the semantic-ui framework for the front-end

feedcloud

The idea is, take an RSS feed, or a list of RSS feeds, and generate a word_cloud for each feed, and the aggregate content from all the feeds, like so:

python gencloud.py config.json

with config files that look like this:

{
    "feeds": [
        "http://blog-decause.rhcloud.com/rss.xml",
        "http://nathanielca.se/feeds/all.atom.xml"
    ],
    "mask_filename": "fossboxlogo-mono.png",
    "output_dir": "feedcloud",
    "output_image": "feedcloud.png",
    "stop_words": ["http", "https"],
    "each_corpi": true,
    "max_words": 1000
}

1. Individual word_clouds for each rss feed AND the aggregate cloud of all posts combined!
2. Uses a config file written in json for easy parsing
3. Once it is set up, it is very easy to create multiple configurations to create many visualizations with ease.
4. easy to use blacklist within config file for easy tweaking.

Demerits:

1. word_cloud is a very powerful, but also very heavy-weight stack to stand up and run. After many attempts, I finally got my first virtualenv working, and I've been using the same one ever since in all my word_cloud related projects :/

wordcloudbot

Arguably the most complete of the tools in the box, it builds upon the work of feedcloud and word_cloud to deliver fancy wordcloud visualizations via Twitter each time an IRC meeting ends and sends the logs message across the fedmsg bus! This tool is "deployed" on my machine locally as a systemd service, but will hopefully be deployed to Fedora Infrastructure in the near-ish future (packaging the word_cloud stack for Fedora is low on the priority totem pole for me personally, but def a goal I'd like to deliver on in the next 120 days (but probably not the next 30/60/90.) You can glance the fruits of wordcloudbot's cycles on the Fedobot twitter account.

fedora-stats-tools

fedora-stats-tools is a breeding ground for interesting one-off scripts and tools. It is the default place where various CommOps experiments are being pushed to, before they end up in their own repositories. Here is what we've got in there so far:

annualgrepper

This is a very basic script that uses the requests library to get the raw json for fedmsgs over the past year, and pulls out the final field of that raw json response, which includes a total number of messages.

meetbot-fedmsg-activity

A jinja2 template that creates links to meetbot activities for each subproject. This is an "artisinal" solution we're using to aggregate datagrepper information prior to the completion of hubs. The challenge is to *not* engineer something, but write as light-weight of a tool as possible to give us raw data. There is also a cronjob running locally on threebean infrastructure to send me a daily reminder to compile the stats from the information gathered by this template.

daily-briefing

A descendant of meetbot-fedmsg-activity, this jinja2 template takes lists of URLs, if they are not empty, and generates a "daily briefing" with things like action items and links generated by meetings and captured by zodbot. Ideally, these lists will be compiled by using BeautifulSoup in the near future, so that every day, the briefing can be shipped without human interaction (and perhaps tweeted by fedobot, but that is another project for another time.)

Long-tail

This particular tool and its purposes was covered at length already in a previous post: http://decausemaker.org/posts/grokkingfedmsg.html

Why a toolbox?

Well, I wanted to include a bit of this in my last post, but for now, here is the gist: The position of Fedora Community Lead has not existed before I had the privilege of filling the slot. There are *many* duties and responsibilities, defined by many stakeholders, and right now, I'm a one-man army looking to deliver on each of them. "Linus does not scale" and neither does decause. I'm working on making each of my contributions "autonomous" so that they do not block on a single person, and can invest the labor needed to deliver on these metrics once, and the deploy/redeploy multiple times as needed. It has been somewhat slow going, but it is my hope that each of these tools can help each problem space or bucket within the CommOps ecosystem of initiatives (see: http://decausemaker.org/posts/proposal-commops-for-fedora.html for the full proposal, and/or contribute your thoughts on the wiki here, https://fedoraproject.org/wiki/CommOps)

Halp!

Metrics is where we've started. It has been a small core of folks working, but the spirit of the Community Operations team should start to become more clear as these types of tools get built. I'm still learning my way around this massive community of FOSS hackers that is Fedora. I still don't know where all of the corners are, or who all the people that make each of those legs gallop with the whole are, but I sure want to. Someday (150ish days from now) there will be FedoraHubs, which is really the ultimate manifestation and incarnation of Community from the amazingly-sleek-and-powerful real-time infrastructure that is powered by the fedmsg ecosystem, but until then, things are going to have to be a bit more manual until we can get there. If you'd like to help tell the story of your particular corner of Fedora, or would like to help provide the "10,000 foot view" on the project's contributor base, then by all means please reach out in IRC (#fedora-apps on freenode) and let me know. I'm here to help, but I sure could use some too :)