Ordinary Skill

This is Part 4 of our series analyzing Federal Court decision turnaround times. Readers should consult Part 1, Part 2, and Part 3 before this one.

In this part, I revisit an idea that was mentioned briefly in earlier editions: how to measure judges’ workloads in comparison to decision-writing time.

We will be looking at cases under reserve (judges’ queue of pending cases) at the Court to see how they may affect wait times experienced by litigants.

All data sources and caveats remain the same as before unless otherwise noted.

Querying the queue

When we previously discussed judicial workloads (Part 1), we looked at the number of cases decided by the Court and by each judge in any given year, which served as a proxy for how busy each judge was. This was a reasonable first pass at analyzing how workload might affect writing time, but it came with some limitations.

In particular, the number of decisions written by a judge is an output of the decision-writing process itself, rather than an input. It is necessarily endogenous, meaning it is not a reliable signal of causality. I also addressed this briefly in Part 3 — given the data available, we do not have many true independent predetermined variables to use as predictors, making it difficult to separate causation from correlation.

After doing some further processing of the data, we can now assess another interesting datapoint, which is the number of decisions under reserve at any given time.

We can measure this as follows: since we have hearing dates and judgment dates, we can determine the number of decisions under reserve for a given judge at a given date by counting the number of decisions that have been heard but not yet issued. For each decision, we can then look backward and see how many other decisions a particular judge had under reserve (or their “queue size”).

In other words, we can calculate how many other decisions need to be dealt with concurrently by the judge at the hearing of any particular matter. The queue size is calculated per-decision-per-judge, so there is a valid “queue size” value for each decision.

Intuitively, queue size is helpful in understanding both a judge’s workload (how many other decisions need to be made at the same time) and on their concurrent cognitive burden.

Queue size is a useful statistical measure because it is independent of case type and output for a given case, and it is a fixed quantity before any given decision is written. Therefore, we can be more confident of the direction of causality. Further, since this metric is quantifiable for each decision, we have a large sample size to work with (n > 10,000).

(Note that this does not yield a “live” calculation of current cases under reserve, since I am using a fixed collection of past decisions from 2016 to 2026 for which we have extractable hearing dates and judgment dates. To avoid artifacts from out-of-range hearing and judgment dates, I have cut off a year from the end of the data.)

Queue size and trends

The queue size of each judge has not changed much over time. On average, each judge has about 4-6 other cases under reserve when they hear a new case, and this has stayed the same for the past 6 or so years. (Figure 1)

Queue size and turnaround time

A visualization of the “queue size” of the hearing judge versus median turnaround time for each decision paints a picture of association between the two variables (Figure 2)

In general, the more cases a judge has under reserve at the time of a hearing, the longer it takes to issue that judgment. This association is highly statistically significant (p < 0.001) across all non-PMNOC cases and has a moderate positive correlation.

PMNOC cases (not shown) are artificially constrained by the need to issue decisions within the 24-month statutory stay and it is not surprising that there was no statistical significance (p=0.832) found against queue size for these cases. This proves a parenthetical point made earlier, in that it appears judges do in fact favour issuance of PMNOC decisions over other cases in their queue, when possible.

The strongest association was found in immigration cases (Spearman ρ = +0.49, a moderately strong positive correlation). An immigration hearing before a judge with 1 other case under reserve at the time of the hearing yields a decision in 8 days, while a hearing before a judge with 11+ cases under reserve takes 99 days on average.

Although this is probably intuitive, we should not necessarily take it as a given.

Inter-judge vs. within-judge queue sizes

One major issue in using queue size across all decisions is that individual judges’ inherent writing style and time is a confounder. If a judge takes longer to write decisions, all else being equal, we should expect that their number of reserved decisions will accumulate, leading to a larger average queue size than for a faster writing judge.

Put another way, one could object that a smaller inter-judge queue size is the result of faster writing, so the observed effect on turnaround time is attributable to the specific judge assigned to the case rather than on how many decisions that judge has under reserve.

However, we can look at the within-judge data to see whether queue size for any given judge is associated with longer writing time, which isolates inter-judge queue-size effects. Below (Table 1) is a table showing the association between queue size and turnaround time for each judge (anonymized and ranked from the highest degree of association to the lowest). The statistically significant values have been bolded.

This can also be visualized in a “volcano plot” as follows, where judges above the dashed line are those meeting statistical significance, and the X-axis (left/right) shows the magnitude of the correlation. (Figure 3)

For just under half the judges on the Court, queue size is a statistically significant positive predictor of turnaround time. For these judges, when their queue is larger on the date of a hearing, they take longer to issue a decision. Conversely, for the other half of the Court, writing time does not seem to vary significantly based on queue size.

This data could result in competing interpretations.

On the one hand, positively, the data support the idea that keeping a “lean queue” demonstrably improves decision efficiency for many judges. The positive correlations among a significant number of judges suggest that there are efficiency gains to be made through active queue management.

On the other hand, the fact that there is no association for half the Court would suggest that inherent writing speed dominates turnaround time for these judges, regardless of queue size.

Indeed, there is a wide variability of “average queue size at hearing” across the bench, as shown in the below box plots. (Figure 4)

The overall unevenness of the judicial queue is a little surprising. There are judges who maintain queues of up to 15-20 cases while others barely have any (it should be noted that some of the judges towards the short end were newly appointed at the end of the data set and thus could not have had much time to accumulate a large queue, and several others at the short end were also supernumerary judges with a lower case load for a material portion of time).

While we do not have much insight into the Court’s internal metrics or docket management, it is likely judges have a lot of independent discretion on how to manage their own dockets and workload.

The variability gap identified here may represent an opportunity. I believe the existence of a correlation between queue size and writing time for some—but not all—judges suggests that there are practices employed by certain judges that could be adopted more widely across the Court. Perhaps additional inter-judge sharing of best practices can reduce variability so that litigants’ wait times are less dependent on the judge assigned to their case.

If this is correct, there can potentially be efficiency gains through more active judicial management and leveling of queue sizes. Tools that are available may include implementing project management practices, providing more real-time insight to the judicial administrator on queue sizes and judicial workload at the time of case assignment, and better load distribution across judges. (It is possible that the Court is already implementing some of these practices, though we do not have much visibility in the Court’s internal operations.)

Summary of factors affecting turnaround time

We have now looked at a number of factors that may affect decision-writing time using publicly available Federal Court data. The dataset allowed us to examine several interesting dimensions, including judicial output, queue size, subject matter, and complexity.

Some of these factors appear to matter more than others. In particular, case complexity and queue size appear to be meaningful (but conditional) predictors of writing time, and we also saw how turnaround time varies by subject matter.

I began this series out of an interest in testing current-generation AI tools to visualize legal data, but the project took on a bit of a life of its own.

To address feedback from various sources, a more rigorous multivariable regression analysis must be done before drawing formal conclusions. That level of analysis goes somewhat beyond the scope of this series, but it would be a useful extension of the work.

I had not originally intended to undertake this analysis – key assumptions and limitations in the data and methodology were set out earlier and, where possible, included statistical significance values (or, just as importantly, the absence of statistical significance was noted) so the results could be interpreted with the appropriate caution.

It bears repeating that the explanatory power of this dataset is necessarily limited by likely confounders and the absence of strong causal controls. As a result, the factors examined here cannot fully explain the observed inter-judge variation in turnaround time.

Nonetheless, for completeness, I also used generative AI to assist with a regression model estimating the relative contribution of each factor. In short, after controlling for the variables available in the dataset, the available predictors account for about 31% of the observed variation in decision turnaround time. The 69% remaining unexplained variation likely reflects a combination of unquantified factors and individual judicial practices or style.

A forest plot of the regression coefficients for each factor is shown below, together with share of variance illustrated, grouped according to type of variable (Figure 5).

In the statistical model used for our data analysis, complexity accounts for about 11% of variability, queue accounts for 19%, other factors account for 1%, and the rest unexplained.

For the sake of completeness (and to anticipate notes from the statistically inclined), I add that it is possible to control for the observed variables (complexity, queue depth, category, and year) to obtain a per-judge residual that would illustrate the difference between expected time based on our modelled predictors versus actual turnaround time, which would represent the inherent inter-judge variability in writing time. However, given that such data is not actionable and because there may be other unanalyzed variables at play, I have not included it here.

Upcoming

Through this now four-part series, we have been able to visualize and analyze some interesting data on decision turnaround times at the Federal Court.

I think we have managed to exhaust most of the available information at this moment (and perhaps more importantly I have probably exhausted the readers). Though there are always more ways to slice the data, further advancement of tools and data availability may enable deeper analytics in the near future.

This series is about to be wrapped up for now. I have only one more part planned, which is a brief look at macro trends rather than specific analysis on turnaround times, so it will be more in the nature of “bonus content” than like the articles so far.

As always, if you have any further ideas or feedback, please share them with me and I will endeavour to respond.