$ cat /var/log/dpkg.log* | grep "\ install\ " | sort

This does a few things:

• First, we concatenate all of the package logs from dpkg, the package manager for Debian. The package logs will have names like /var/log/dpkg.log.1, /var/log/dpkg.log.2, and so on. (The oldest logs will be zipped up, using Ubuntu's built-in log rotation, so they will look like /var/log/dpkg.log.3.tar.gz.)

• This is then piped to grep so that we find only installation events. Specifically, we look only for lines that contain a space, followed by the word "install", followed by another space.

• Finally, we sort the list of items we got. Because each dpkg record line begins with a timestamp, items that were installed earlier will be sorted first in this list.

The result is a chronological list of package installation events, with the most recently installed package first. (Note that if a package was installed and subsequently removed, it will still be on the list.)

If you want to get some subset with the most recently installed one first, you can use a combination of sort -r and head -n. For example:

# Retrieve the last 5 most recently installed packages, with
# the most recently installed one first.
$ cat /var/log/dpkg.log* | grep "\ install\ " | sort -r | head -n 5
2009-06-20 23:47:49 install easytag <none> 2.1.4-1.1
2009-06-20 23:47:49 install libid3-3.8.3c2a <none> 3.8.3-7.2
2009-06-19 18:08:02 install wv <none> 1.2.4-2ubuntu2
2009-06-19 18:08:01 install tracker-utils <none> 0.6.93-0ubuntu3
2009-06-19 18:08:01 install tracker-search-tool <none> 0.6.93-0ubuntu3

Update: Commenter maximd noted that you can also bring zcat into the equation to get an even more comprehensive list that will introspect the log files. This will hit the older logs that have been zipped up, so you can go farther back if needed.

As committed, the code looks like this:

public static IEnumerable<IEnumerable<TSource>>
  ContiguousSubsequences<TSource>(this IEnumerable<TSource> self, int windowSize)
{
  Check.Self (self);
  if (windowSize < 1)
    throw new ArgumentOutOfRangeException("windowSize", "must be >= 1");

  return CreateContiguousSubsequencesIterator (self, windowSize);
}

private static IEnumerable<IEnumerable<T>>
  CreateContiguousSubsequencesIterator<T>(this IEnumerable<T> input, int windowSize)
{
  int index = 0;
  var window = new List<T>(windowSize);
  window.AddRange (new T[windowSize]);
  foreach (var item in input) {
    bool initializing = index < windowSize;

    if (!initializing) {
      window = window.Skip (1).ToList ();
      window.Add (default (T));
    }

    int itemIndex = initializing ? index : windowSize - 1;
    window [itemIndex] = item;

    index++;
    bool initialized = index >= windowSize;
    if (initialized)
      yield return new List<T>(window);
  }
}

>>> a = 500
>>> b = 500
>>> a is b
False

>>> c = 200
>>> d = 200
>>> c is d
True

In Python, is tests for identity, not equality. x is y if and only if x and y reference the same thing. Although a and b have the same value, they are distinct objects, and so comparing the two yields False, as one might expect.

But then we're confronted with the second case. It's precisely identical to the first, just with a different assigned value. Yet it produces the opposite result. How can this be?

The key to this puzzle lies in a peculiar implementation detail of CPython, the de facto Python implementation. As we said earlier, in Python, everything is an object, even literals. Logically, that means that two different instances should be distinct from each other, as in the first case above.

But in CPython, when you create an integer literal in the range [-5, ..., 256], it's actually cached for performance reasons. Further references to the same literal are identical references to the existing literal, not new references. Thus c and d refer to the same cached instance, and the result is True.

Because of another implementation detail, two literals with the same value that are in the same compilation unit will reference the same object. Comparing literals directly results in True in both cases, as we see here:

>>> 200 is 200
True

>>> 500 is 500
True

More importantly, however, this illustrates the danger when is is mistakenly used to compare value equality instead of reference equality. Had you used == instead, the results are precisely what you'd expect:

>>> a = 500
>>> b = 500
>>> a == b
True

>>> c = 200
>>> d = 200
>>> c == d
True

$ curl -s http://checkip.dyndns.org | sed 's/[^0-9.]//g'
209.145.68.249

This is a simple one-two punch. First, we use the curl tool to get back a HTML document that contains our IP, using the checkip service from DynDNS. Next, we need to filter the output to return just the IP address.

Then we pipe the output to sed, with the instructions s/[^0-9.]//g. This tells sed to do the following:

• invoke the (s)ubstitute command;
• find the first character ([...]) which is not ([^...]) a digit (0-9) or a period (.);
• replace each such character with an empty string, effectively removing it;
• repeat this command (g)lobally for the entire stream

The result is our IP address.

I was pleased at how remarkably easy it was, especially since I was a little nervous about touching code I hadn't seen in a while. As for whether this means that Periscope is well-designed or if it's just those energy drinks temporarily increasing my focus -- well, I'll leave that for you to decide.

But for other curious hackers, the special sauce is in the handler map:

// For each domain, we provide a domain handler that
// understands how to provide a meaningful title for the
// periscoped link. We optionally provide an id handler that
// knows how to transform the identifier of each periscoped
// link so that it's easier to find what we're looking for.

// [other handlers elided]
handlerMap["is.gd"]          = {domainHandler: handleIsGd, idHandler: null};
handlerMap["bit.ly"]         = {domainHandler: handleGetTitle, idHandler: null};
handlerMap[""]               = {domainHandler: handleUnknown, idHandler: null};

Each of the handlerMap lines maps a particular domain name identified from one of the document's links. In this case, when you mouseover a particular bit.ly link, you'll now see the additional information provided by the handleGetTitle handler. Hovering over, say, http://bit.ly/GH4Cn thus lets you know that this goes to Google's search page.

In a future article I'll show how to write your own handlers. Thanks again to everyone at beCamp2009 for attending my talk. If you have any questions, feel free to drop me a line using my contact page.

Justin Etheredge drew my attention to a recent article in Artima Developer entitled Twitter on Scala, describing some issues that Twitter encountered as it scaled up, and providing justifications for switching the codebase to a different language. While that is interesting in and of itself, what was really thought-provoking is the excerpt from Alex Payne that Justin cites, reproduced in its entirety here:

Alex Payne: I'd definitely want to hammer home what Steve said about typing. As our system has grown, a lot of the logic in our Ruby system sort of replicates a type system, either in our unit tests or as validations on models. I think it may just be a property of large systems in dynamic languages, that eventually you end up rewriting your own type system, and you sort of do it badly. You're checking for null values all over the place. There's lots of calls to Ruby's kind_of? method, which asks, "Is this a kind of User object? Because that's what we're expecting. If we don't get that, this is going to explode." It is a shame to have to write all that when there is a solution that has existed in the world of programming languages for decades now.

– Twitter on Scala. Artima Developer. Published April 3, 2009. Accessed April 7, 2009.

Is Alex unfairly scapegoating Ruby for Twitter's failings, and using this as a reason to move to Scala? Obie Fernandez seems to think so. If the kind_of? checks need to be proliferated because Ruby has framework and language weaknesses, that's one thing. But if it's Twitter legacy code, that's another matter entirely.

In a Twitter exchange between Alex and Obie, Alex admits the problems were with Twitter, not Ruby:

[Alex Payne] » @[Obie Fernandez] Indeed, using kind_of? in that way is "doing it wrong" in Ruby. But as our codebase grew, it became a necessity to combat bugs.

[Obie Fernandez] » @[Alex Payne] Doesn't make sense. Do you mean non-deterministic bugs? An in-depth explanation of the bugs you're talking about would be enlightening

[Alex Payne] » @[Obie Fernandez] Yes, I mean non-deterministic bugs in the giant, legacy, spaghetti parts of our system. Unexpected objects flying around.

It's not very fair to be armchair architects and second-guess Twitter's design decisions with the benefit of hindsight. Nonetheless, I'd be willing to bet that a more thorough analysis of the code base would have revealed the source of the "unexpected objects flying around". That is likely to be the real cause of their issues, not the proliferation of ad-hoc type checks.

But are these perhaps part and parcel of the same underlying problem?

Catching problems

We've stumbled onto an interesting question: Does the use of dynamic type systems defer some of the responsibility that would otherwise be handled by static type checking to the developer in the form of additional required tests? There's no doubt that this is the case. Alex's experience with the Twitter codebase is compelling (albeit anecdotal) evidence for this point.

To see why, consider that the underlying goal of a static type system is to convert as many programming mistakes as possible into type mismatch errors (TME). A TME occurs whenever an expression needs to be of some type T, but its static type is of type U, and there is no automatic way to convert U into T.

Possible TMEs are easy to identify statically. But most static languages will be conservative and reject programs which wouldn't have any issues at runtime. In other words, merely having a candidate for TME can be sufficient to cause a failed compile. For example, the C# compiler will reject a snippet such as the following:

int x = 0;
int y = 1;

if (y == 0) {
  x = "hello!";  // Won't compile, even though it's impossible to reach
}                // this line at runtime and no type error can occur.

In the above example, the attempt to assign "hello!" to x will be a TME: an assignment to x for a static type is required to be of the same type as x or of a type that can be implicitly converted to x. In this case, x is an int, and the expression "hello!" is a string. C# has no implicit conversion from a string to an integer, so failure occurs.

But on closer inspection, we see that this line could never be reached to begin with: the y == 0 condition cannot possibly be true. We're often told to trust the compiler, and let it do its job, because understanding compiler internals and optimizations seems to be trickier than just writing code. But this example seems to go against that; it's pretty clear that this wouldn't really be a problem. Why can't the compiler be smarter than a human in this almost absurdly trivial case?

A tradeoff: flexibility of expression for type safety

The answer is that static type systems try to be as strict as possible to make their analysis easier. In exchange for this strictness, static compilers can make more powerful guarantees about the type safety of their programs at runtime.

Additionally, an assignment like int x = "hello!"; might be considered wrong on its face, since there's a mismatch between what was intended and what was stated. By highlighting such errors at compile time, where they're more easily caught, we save a lot of developer time later down the road. The example we used here is trivial to the point of being a straw man, but in more complex expressions it may not be automatically obvious to a human whether the result is what was intended.

Lost type safety might need to be compensated for elsewhere

Many dynamic type systems give us more powerful constructs to express software than static typing does. It's often quicker to just say what you mean in Python, Groovy, or Ruby and have unit tests sort out the details later.

But when you get around to writing those tests, what do you actually need to test? Certainly, the tests you write for a statically typed language will be similar in some respects to dynamically typed ones. You'll still need to test the logical portions of your software -- for example, whether ComputeSalesTax() computes sales tax correctly, or whether a particular Comparable mixin sorts instances the way you expect. So we wouldn't expect to see much difference between unit tests of static and dynamic languages on this front.

But there's some additional work you need to do with the unit tests of dynamic languages. In general, without checking things yourself, you can't be sure at runtime that a particular object will respond to a particular method signature. More generally, you can't know in advance that it conforms to a particular interface.

That means that, if you really do require the full power of static typing in a dynamic language, you will unquestionably need to do some more legwork in testing to be sure that you didn't miss anything. Each and every time you invoke a method, for example, you will need to scrupulously examine its arguments for compatibility with the method signature. In effect, you will have ultimately implemented static typing for the subset of your software being tested, and in a far more verbose (and likely error-prone) way¹.

This should not be a surprising result: if you want to implement a feature of language L in a different language M using only the provided facilities of M, be prepared for an uphill struggle. This is why, for instance, no one has been successful in getting STM into widespread, mainstream languages: they're not well suited for it.

The real point here is that if you need static typing, you should use a statically typed language. Similarly, if you absolutely need native concurrency primitives, you don't implement them in C#; you use Erlang. Static typing is no different from other language features in this respect.

Best of both worlds: opt-in type systems

As developers, that's frustrating to hear, of course. Why can't we have our cake and eat it too? What if we want the syntactical conveniences of Ruby with the static typing of Java?²

Fortunately, recent additions to mainstream programming languages are beginning to make this sort of thing a reality. For instance, C# 4 will soon have the much-vaunted dynamic pseudokeyword. This essentially functions as a type modifier; expressions of type dynamic have their evaluation deferred to runtime. Effectively, C# now has what I'll call opt-in duck typing.

Even dynamic languages have toyed with the notion of batting for the other team. Guido van Rossum has entertained the idea of static typing in Python at least once before. And ActionScript 3 has optional static typing, an obvious complement to C#'s approach.

You may be able to see how this feature would be valuable in some circumstances -- perhaps you can even identify situations where it'd be useful in your current project. But more importantly, were you to use these opt-in type systems, your unit tests would change accordingly. A dynamic language using opt-in static typing would see some manual type checking disappear for the relevant unit tests, and likewise a static language using opt-in dynamic typing might need a few more checks to make sure messages were going around correctly.

Dynamic unit tests represent your elective use of typing

Dynamic unit tests, then, are like a barometer for measuring how much you care about typing in a particular segment of your software. If you're relatively indifferent to type so long as objects conform to some interface (which is generally the case), then your unit tests will reflect this. This is often the case, which explains why dynamic languages can feel so productive and baggage-free.

But if you'll have difficulty unless the types are exactly right, or if things don't line up just the way you expect, your unit tests will be much richer in assertions and checks. They're essentially functioning as a substitute for as-yet-nonexistent language features like opt-in alternative typing systems.

That, I would surmise, is why Alex and the Twitter crew found themselves a new home with Scala. It wasn't that Ruby was a poor language or that it was ill-suited for development of large-scale software. It's that Scala offered the static typing and rigor that was needed for their particular application. It allowed them to transform those pervasive ad-hoc type checks and rely on the language to provide a more stable type system, one where their assertions could manifest as type declarations instead of ad-hoc typing in unit tests.

At the end of the day, a better product resulted, and that's the goal. Our choice of which language features to use is only tangentially related to the quality of software we produce. But if we can make it easier on everybody with elective language features, and avoid the proliferation of checks that arise when these aren't available, so much the better.

Footnotes

¹ This of course says nothing about the quality of software produced with dynamic or static languages. That has, in my view, less to do with languages and more to do with design decisions and disciplined testing.

² This is really a manifestation of a sort of 80/20 rule: we're getting most of our day-to-day language productivity directly from a relatively small subset of language features. The frustration arises because we want to do that last 20%, but it'd require clumsily accreting stuff from the remaining 80%.

http://www.codethinked.com/post/2009/04/05/Do-We-Create-Type-Systems-In-Dynamic-Languages.aspx