Mediargus Content Acquisition

Mediargus, the Belgian press-monitoring company, collected articles from newspapers, magazines, news agencies and websites every day so its customers could search and follow them. From February 2011 into 2012 I built and kept extending most of the importers that brought new sources into Mediargus, each turning a source’s own format into Mediargus’s XML article format, plus parts of the services and the internal admin tool around them. Each source got its own small repository, sharing the same structure and contracts.

News sites over RSS. For De Wereld Morgen, StampMedia and IPS, the importer read the source’s RSS feed and handed every article to a pool of worker threads while it was still parsing the rest of the feed. The workers cleaned up the article’s HTML down to its text, links and embedded media, converted it to Mediargus XML, and skipped anything already seen: every article got a SHA-512 fingerprint of its content, stored in MySQL, so a feed could be read as often as needed without creating duplicates. The tests ran against the real feeds and a staging copy of De Wereld Morgen.

Agency and newspaper deliveries. Other sources delivered files instead. ANP sent news agency articles as NewsML and NITF files, which were converted, backed up and cleared from the input folder. Het Nieuwsblad delivered every regional edition as its own folder with page PDFs and article XML; the importer mapped each edition and region to Mediargus’s own publications through configuration, transformed the articles with XSLT, and filed the page PDFs by date, publication, edition and page. For The Bulletin, the English-language weekly, it split the full-edition PDF into one PDF per page. I also worked with Geert Van Huychem on the importer for HLN’s zipped deliveries, with its zones, pages, articles and images.

Following Twitter. In August 2011 I built the first version of a social dashboard that followed Twitter for Mediargus users. Every user could keep a list of search terms, and the monitor followed them live on Twitter’s streaming API, while a second stream, signed in with the user’s own Twitter account over OAuth, followed their own timeline. Both connections stayed open around the clock and reconnected on their own, backing off as Twitter’s guidelines asked: linearly after a network error, exponentially after an HTTP error.

Services for the new site. Alongside the importers I worked on Mediargus’s WCF service layer: every service exposed both SOAP and JSON endpoints, with a unit of work per call over the old Oracle database and a new MySQL one. It handled guest accounts that people could link to their Twitter, Facebook or LinkedIn profile, with a verification mail in Dutch, French, English and German, tagged articles and other content, and handed out blocks of new article ids to the importers.

An admin dashboard. For the support team I built an internal ASP.NET MVC dashboard to handle the common requests without touching the database by hand: restoring deleted users, giving a user access to extra publications, resetting a customer’s prepaid balance, and looking up the IP ranges that public libraries used to reach Mediargus.

Technologies: C# 4.0, .NET Framework 4.0, WCF, ASP.NET MVC 3, Razor, jQuery, DataTables, NHibernate, Fluent NHibernate, Oracle, MySQL, StructureMap, Castle DictionaryAdapter, log4net, NUnit, Rhino Mocks, RSS Bandit, HtmlAgilityPack, XSLT, PdfSharp, SharpZipLib, Twitter Streaming API, OAuth, Twitterizer, Subversion.

David is a great colleague to work with, packed with excellent coding skills in the .NET framework, but at the same time keeping an open mind, very pragmatic and a teamplayer. A real asset to my team.

David is a creative thinker, backed up with detailed knowledge of the possible means to achieve the desired results.