I couldn’t have been more mistaken. NPR’s API is no small potatoes. Just take a look at the comprehensive Query Generator to get an inkling of the types of complex queries you can create. Looking at the Query Generator also sheds some light on the content you can retrieve using the API. The API’s main page says the API exposes the entire NPR archive of content starting from the launch of the NPR website in 1995. Just how big is this archive? Over 250,000 stories including text, images, video, and audio!

This quote from the article announcing the API caught my eye immediately:

There were quite a few questions that we addressed when developing the API, but one thing that was not really in question was the need to open as much of our content as possible.¹

This isn’t the first open media API. BBC was the first to offer a public open access API, however BBC’s API is restricted to the content from the past 7-days. Seven days! That’s nothing compared to the (approx.) 4748 days - and counting - that NPR’s API offers. NPR and the BBC are two large companies leading the technological shift towards open and free information.

But that’s only half the story.

After discovering this fantastic API I had to do something with it, and the new service architecture in Amarok 2 provided the perfect platform to build a NPR mashup. That was several months ago, and at the time the scripting API in Amarok was still being flesh out (Thanks to Peter). On Monday I noticed the BBC scriptable service Nikolaj had created for Amarok 2. I happened to have several hours of free time, so I cooked up a similar service for NPR:

You can get it at kde-apps or via the “Get More Scripts” button in Amarok 2’s Script Manager.

There is definitely room for improvement and in fact here are a few things I plan to do with it:

• Display more than 20 stories under a category
• Sort content by date
• Support searching
• Display the full articles, with images, in the context view

Major props and thanks go out to the entire NPR technical team and all the contributors who made API a reality.

1. NPR API is Live on NPR.org

So, lets see… last time I posted I was en route to Akademy 2008 (the KDE developers conference). That was July 30th, now, a month and a half later I am back from Europe (which was amazing), GSoC is over (sad), and class has started (jury’s still out).

Current Status of the MP3tunes Amarok Service

The Good (Works)

• Browsing & Streaming
• Querymaker is as functional as possible with the current API
• Manual Downloading from MP3tunes to Local Collection
• Manual Uploading from any Collection to MP3tunes

The Bad (Doesn’t work)

• AutoSync - 80%- The code is there, however there are problems with the daemon not receiving signals from the MP3tunes’ servers
• MP3tunes playlists support - 0%- Never got started on this

The Ugly

There is one large issue I am still wrestling with. The details of the issue are complicated, but essentially it deals with the way Amarok handles remote tracks in playlists across sessions. Generally, remote tracks’ metadata isn’t cached by Amarok for use between multiple sessions. So, if you add an mp3tunes track to a playlist in Amarok, then restart Amarok and try to play that playlist the mp3tunes track is blank. Now, there exists a method for retrieving that metadata, however it was originally implemented synchronously. When you are retrieving metadata for any significant number of tracks (10+) synchronously, and each one of those retreivals is an HTTP Get request you end up blocking the GUI thread. At Akademy I hacked a way to do this asynchronously, but it is a really nasty hack. I’ve got code on my computer that implements this feature correctly, but it has the nasty habit of crashing Amarok every so often.

Between classes, marching band, homework, and other responsibilities I’m working on getting this ironed out and committed. Hopefully this will happen before the 2.0 release, because right now using MP3tunes in Amarok across sessions is slow and annoying.

• clothes
• 2 laptops
• 1 Target desk (retail $50)
• assorted books
• 1 blow-up air mattress

Fast forward through seven hours of me hurtling down the interstate at not-so-safe velocities, and here I am, pardoning my recent idleness as my flight to Paris boards at gate D32. Not accomplishing much over the past several weeks suddenly doesn’t seem so bad: I’m going to Europe! There is a week long hack-a-thon at Akademy; I’ll catch up then.

A bientôt!

For the past week I’ve been banging my head against the wall of glib, QtEventLoop, and QThreads. I have quite a headache to say the least, but yesterday thanks to my mentor and Ian, both Amarok developers, my head actually broke through that wall. Literally. GLIB, and Qt are kowtowing at my feet swearing oaths of fealty. They have promised to work together and let me get back to doing fun things, like code new features.

Surprisingly, bulleted lists seem to be the most concise means to list progress [/sarcasm]

• Still working on Harmony integration.
• Harmony authentication is working (Thanks Lateralus from MP3tunes!).
• The daemon is receiving notification updates.
• Notification processing is what I’m working on now.

While looking at my last couple reports about harmony, I realized that the information is somewhat dry. I think this is because harmony is a highly technical, behind the scenes feature, and, also, often it is difficult to inject excitement into mundane programming updates. The humdrum is slightly symbolic of harmony. You will enable it once and forget about it as it silently keeps your music collection synced.

Related posts: GSoC Report Week 5: Harmonizing Amarok, One small step for Amarok…

Total Commits: 106 Weekly Commits: 26

Past 7 Days

I usually aim to post these reports on Monday, but I’m usually wrapping up a final commit or two on Monday night so I wait till Tuesday to post the report. Well now it’s 3:30 on Wednesday morning and I’m just starting. My last commit for “this week” was, oh, about 30 seconds ago.

During the last 7 days I:

• Patched libmp3tunes to support track fetching based off a filekey.
• Used the aforementioned patch in Amarok to enable saving of MP3tunes tracks to playlists.
• Made the MP3tunes Service “Lazy Load” upon Amarok’s start-up.
• Fixed misc non-mp3tunes related Amarok bugs.
• Added libmp3tunes::Harmony to the source tree.
• Created a harmony daemon that runs asynchronously within Amarok.

As usual you can see a list of my most recent commits via my fisheye page.

What the heck is this Harmony nonsense?

Harmony, itself, is a subset of libmp3tunes that provides an api for receiving event notifications from the MP3tunes servers. Essentially, what it boils down to is harmony enables the MP3tunes servers to notify Amarok when a user’s Locker has been changed. This will allow Amarok to assess the changes and perform an appropriate action (e.g., download a new track).

Suppose Jenni buys a song from eClassical and has it loaded directly to her locker. When this happens her Amarok will receive a notification: “Hey Amarok, Jenni just had a track added to her locker.” At which point Amarok will seamlessly download the track to Jenni’s local collection.

Upcoming 7 Days

Pretty cool right? Sure is, there’s just one caveat: it’s not working yet. Getting harmony to play nice with Amarok was a challenge that took a couple days, but as of this morning harmony is running in Amarok.

By this time next week I plan to have harmony fully integrated with Amarok, so the above scenario can actually take place. Even though the feature freeze that was announced for Amarok 2.0 technically doesn’t include me, I will still be taking some time to polish all the work I’ve done since May.

Total Commits: 84 Weekly Commits: 36

Past 7 Days

Starting these posts with “It was another busy week..” is starting to get boring; I’ll cook up something more exciting for next week.

In case you missed it, in the past seven days history has been made. I’ll let that stand in as the bulk of my weekly report, but a few worthwhile things have occurred since then that deserve a mention.

Remote Track Upload - You can now sideload tracks to your MP3tunes collection from remote sources in Amarok. What the heck is sideload and what remote sources you ask? Sideload is a feature of the MP3tunes API that allows for server-to-server transfers. This means you can give your Locker a URL to a track, and it will automatically be downloaded into your Locker. Currently Amarok sports three services with remote collections that are sideloadable to MP3tunes: Ampache, Magnatune, and Jamendo. This method of transferring is generally very fast, because the transfer bypasses your slow internet connection.

Upload Progress Bar - When you upload (or sideload) tracks to MP3tunes there is now a simple status bar to let you know how far along in the process you are.

Non-Supported Track Filtering - This one is simple: If you try and upload a file-type that MP3tunes doesn’t support, Amarok will tell you and stop that track from being transferred.

Upcoming 7 Days

• Work on allowing MP3tunes tracks to persist after a restart.
• Flesh out the synchronization system: what it’s going to do, and how it’s going to do it.
• Design any UI widgets needed for the syncing system

The first one will be simple, in fact I plan to code it up after writing this report. These second and third tasks, however, signify that I’m moving into the last stage of the project. According to my original proposal I am ahead by a week, so I’ll be using this week as extra time to plan and get a jump-start on the synchronization framework.

This lucky track happened to be Making Me Nervous by Brad Sucks available over at the great indie music label Magnatune.

Early this morning I committed the last bit of code that allows you to upload tracks to your MP3tunes Locker from Amarok. Amarok is the first client, besides the official client, to allow you to do such a thing. One of the great things about this feature is it’s seamless integration in the UI.

Click for the full view

After a short upload I go check the MP3tunes Web Player

From this point I can listen to the track on my phone, on my squeezebox, or any other MP3tunes supported devices.

Of course there are some caveats, but I aim to fix these over the next several days:

• No upload progress information.
• No error handling when you attempt to upload a filetype mp3tunes doesn’t support.
• It’s not possible to upload non-local content (say from magnatune or ampache)

Don’t start thinking that things are winding down; uploading and downloading are only one small part of the show. The end goal is fully automated bi-directional syncing between Amarok and MP3tunes with 100% support for the MP3tunes API.

What exactly does this entail?

1. I foresee a “keep in sync with MP3tunes” checkbox for each playlist in the Amarok playlist browser, so you can add/remove tracks from your favorite playlists and thoughtlessly have access to them on any MP3tunes enabled device.
2. Imagine clicking “Purchase” at an online music store (like eclassical) and having the tracks instantly available in Amarok and anywhere else you have access to your locker.
3. Your friend beams you a track from his Android phone to your Android phone while you are out on the town, and when you get home the track is all ready in your local Amarok collection, added to your smart playlists ready to jam.
4. You subscribe to an awesome podcast using Amarok’s built in podcast directory, and seconds after a new episode is released it’s available on your wifi enabled portable player (or phone)

And it all started today.

Total Commits: 51 Weekly Commits: 12

Past 7 Days

It was another productive week in #amarok with over 150 commits! In the 12 of those that were mine I managed to do several things.

• Added elegant session handling to the MP3tunes service
• Fixed the collection search box, so you can filter your MP3tunes Locker.
• Enabled Copy To Collection functionality, so you can now copy (read: download) mp3tunes tracks to some other collection!

Of course, implementing those items wasn’t as simple as it sounds, but the features are essential and basic.

The search box (filtering) could use some improvement as currently it only filters via the artist field, but that is a limitation of the MP3tunes API. When I say it
“only filters via the artist field” I mean that it only matches against artists, so searching for a particular track name will not work. To fully support the filtering feature the MP3tunes API would need to allow you to do something like get a list of artists based on a partial track name in a single request. That is just one example, and yes, I could workaround it by doing multiple queries, however that would slow the entire operation significantly. Users expect the search fields in Amarok to be snappy, not take ~5 seconds per. token they supply. It is certainly not a showstopper, and it functions well enough for now, but hopefully MP3tunes will be open to expanding their API later on down the rode. To be fair I have never come across a web API that supported that sort of complex searching. The Ampache service in Amarok suffers from the same lack of functionality.

Interjection: Major props to my GSoC mentor, Nikolaj, for attempting to explain various parts of Amarok’s innards to me, not only once, but the several times it took to get the concepts through my thick skull. Also, he’s helped me track down several childish mistakes I’ve made when I was at my wits end trying to locate them. I can’t thank him enough. Hands down he’s the best GSoC mentor.

With the addition of “Copy to Collection” Amarok has taken a large step towards being fully integrated with MP3tunes. Up till this week all you could do was browse and stream your MP3tunes Locker. That is fine and dandy, but you could do that from the MP3tunes web player, their mobile player, your PS3, or any other number of their supported devices. However, none of those options allow you to seamlessly download and organize your stored music into your local music collection at the click of a button.

There is one shortcoming that needs to be addressed at some point before I’m satisfied: there is no progress indicator of any kind when you download tracks. The only way to see if tracks are being downloaded after you press Go is to watch the destination directory for changes. Thankfully this affects all collections you can “copy to/from”, not just MP3tunes, so perhaps someone else will feel inclined to whip up a progress indicator. There’s no such thing as a free lunch.

Upcoming 7 Days:

I have one big goal this week:

• Add MP3tunes Upload features

By Monday next week, you will be able to do Copy tracks from your local collection, Ampache collection, and the Magnatune database, to your MP3tunes locker.

There is quite a bit of work to be done before this can happen, but I will spare you the gritty implementation details until next week after I’ve committed the code where my mouth is (?).

Of course my weekly predictions wouldn’t be complete without a task to fall back on if I happen to complete the aforementioned task in a Ballmer-Fueled rage. After upload is in place there is only one major item left: Syncing. I need to break “Syncing” into manageable actionables (quite a term, eh?) and then lay out some mid-level designs for the process. Later on during the week I will dedicate an entire post to this topic.

Total Commits: 35 Weekly Commits: 15

Past 7 Days

This was a busy week. I migrated the existing MP3tunes service in Amarok 2 from making REST calls and parsing XML manually to use libmp3tunes. I created an object oriented encapsulation framework in C++ for libmp3tunes, which is written entirely in c. This means instead of mucking about crafting http queries and worrying about parsing data from XML one can manipulate the Locker in an OO fasion. Here’s a little snippet of how libmp3tunes saves work.

Without libmp3tunes if you wanted to fetch a list of artists this is what it would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

    QString urlString = "http://ws.mp3tunes.com/api/v1/lockerData?sid=<SESSION_ID>&partner_token=<PARTNER_TOKEN>&output=xml&type=artist";
 
    urlString = "http://ws.mp3tunes.com/api/v1/lockerSearch?output=xml&sid=<SESSION_ID>&partner_token=<PARTNER_TOKEN>&type=artist&s=" + m_artistFilter;
 
    urlString.replace( "<SESSION_ID>", m_sessionId);
    urlString.replace( "<PARTNER_TOKEN>", CENSORED);
 
    debug() << "url: " << urlString;
 
    m_storedTransferJob =  KIO::storedGet(  KUrl( urlString ), KIO::NoReload, KIO::HideProgressInfo );
        connect( m_storedTransferJob, SIGNAL( result( KJob * ) )
            , this, SLOT( artistDownloadComplete( KJob *) ) );
 
    ArtistList artists;
 
     //so lets figure out what we got here:
    QDomDocument doc( "reply" );
    doc.setContent( m_storedTransferJob->data() );
    QDomElement root = doc.firstChildElement( "mp3tunes" );
    root = root.firstChildElement( "artistList" );
 
    QDomNode n = root.firstChild();
    while( !n.isNull() )
    {
        QDomElement e = n.toElement(); // try to convert the node to an element.
 
        QDomElement element = n.firstChildElement("artistName");
        ServiceArtist * artist = new ServiceArtist( element.text() );
 
        element = n.firstChildElement("artistId");
        artist->setId( element.text().toInt() );
 
        ArtistPtr artistPtr( artist );
 
        artists.push_back( artistPtr );
 
        m_collection->acquireWriteLock();
        m_collection->addArtist( artistPtr );
        m_collection->releaseLock();
 
        n = n.nextSibling();
    }

With libmp3tunes it looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

    Mp3tunesArtistFetcher * artistFetcher = new Mp3tunesArtistFetcher( m_locker );
    connect( artistFetcher, SIGNAL( artistsFetched( QList<Mp3tunesLockerArtist> ) ), this, SLOT( artistDownloadComplete( QList<Mp3tunesLockerArtist> ) ) );
 
    ThreadWeaver::Weaver::instance()->enqueue( artistFetcher );
 
    ArtistList artists;
 
    foreach(Mp3tunesLockerArtist artist, artistList) {
        ServiceArtist * serviceArtist = new ServiceArtist( artist.artistName() );
 
        serviceArtist->setId( artist.artistId() );
 
        ArtistPtr artistPtr( serviceArtist );
 
        artists.push_back( artistPtr );
 
        m_collection->acquireWriteLock();
        m_collection->addArtist( artistPtr );
        m_collection->releaseLock();
 
    }

Both of those code samples produce this:

Notice that instead of looping through XML and ripping out data, I was able to call getter methods to retrieve the same data. Of course the XML parsing has only been moved to libmp3tunes, but by hiding the MP3tunes API implementation from Amarok it creates more maintainable code.

If none of that made much sense, no worries, the important bit to grasp is that libmp3tunes does these important things:

• Encapsulates the MP3tunes API so it is separate from the rest of Amarok.
• Provides an Object Oriented interface to the mp3tunes API. Creating a new session is as easy as Mp3TunesLocker locker = new Mp3tunesLocker();
• Ensures Amarok is officially supported by mp3tunes as long as they support libmp3tunes.

Upcoming 7 Days:

There are a few libmp3tunes shortcomings. One is the lack of a means to detect when a session has expired. Each MP3tunes API request requires a valid session (except of course the initial session-establishing request), and each session times out eventually. When using MP3tunes in Amarok it will be important to elegantly handle session timeouts, for the user does not care about sessions or timeouts. When the user clicks play on an artist they expect it to play, while currently, if the session has timed out Amarok doesn’t do anything. This week I will patch libmp3tunes to support detection of timed out sessions.

Another goal for this week is to fix the search box, so it actually searches.

Also, if you right click on an artist in the MP3tunes collection browser you get a “Copy to Collection” option. At the moment it doesn’t do anything. After this week is over, hopefully, selecting the “Copy to Collection” option will let you do just that.

Foreseeable Roadblocks:

Last week’s roadblock still stands. In the next few weeks I’ll be getting closer to the time when I will need to implement that syncing part of libmp3tunes into Amarok. The licensing issue won’t stop me from developing it on my own workstation of course, but it will have to be resolved before I can commit that part of library or code that implements it.