2016 World Class Index: Where Does Greater Philadelphia Stand?
As the Economy League releases the 2016 annual update of the World Class Index, we’ve come to understand and appreciate the fact that tracking metropolitan progress is not a straightforward endeavor. Change is the only constant.
With three indices in the books, we have seen how focused efforts are driving progress in our region with a number of promising trends across education and talent development, business growth, and infrastructure. In the past year, employment growth has accelerated markedly; the on-time graduation rate in the city of Philadelphia is on the rise; and new businesses are expanding. Throughout the fall and winter, the Economy League will monitor these trends and provide deeper analysis here at Regional Direction.
At the same time, modifications to data collection and reporting – sometimes due to political or budgetary forces, sometimes due to changes or improvements in methodology, sometimes due to a lack of sustained interest in a topic – can make it challenging to understand trends. While identifying a static set of indicators seems simple enough on paper, in reality the way that data is measured and collected changes often.
For the 2016 Index, changes in data availability altered how we measured several indicators, particularly for K-12 student achievement – with the introduction of new exams – and for household economic indicators as a result of budgetary cuts at the Census.
New School Exams Across the Region
All three states in our region introduced new student assessment exams for the 2014-2015 school year to adapt to the Common Core or updated state standards. The new exams in New Jersey and Delaware – Partnership for Assessment of Readiness for College and Careers (PARCC) and Smarter Balanced, respectively – are part of multi-state testing consortiums, while the revised Pennsylvania System of School Assessment (PSSA) and Keystone Exam reflect more rigorous Pennsylvania Core standards. While Delaware has been using the SAT to measure performance of its 11th graders for a number of years, the SAT itself was also revised in 2016 to align more closely with Common Core standards.
These changes affect indicators on early literacy and college and career readiness. Although we had never been able to compare across the states in our region because of differing exams, the change in assessments means we also cannot compare to historical performance within each state. In New Jersey, for example, both statewide and regional proficiency scores in 2015 were significantly lower than scores in 2014 on the High School Proficiency Assessment (HSPA), but this does not necessarily reflect poorer student performance. Standardized tests evolve annually in their content, testing method, and grading, so year-to-year comparisons have always been a challenge. As a result, our best option continues to be comparing students’ performance in each Greater Philadelphia sub-region (Southeastern PA counties, Southern NJ counties, and New Castle County) to students’ performance in each state in a given year, and we have reset our comparative analysis of student progress year over year.
The introduction of new exams reflects continuous efforts to improve how student performance is measured and to meet more rigorous state and national standards. However, the frequency of these exam changes generates the unwelcome byproduct of eliminating the ability to clearly understand trends in student performance over time, thereby making it difficult to assess the impact of new academic approaches.
Changes at the Census
At the federal level, budget cuts to the U.S. Census Bureau in recent years have resulted in reduced availability of data for the 2016 Index. The American Community Survey (ACS) is the go-to source for a number of indicators on demographics and economic status including educational attainment, poverty rates, and household income. The ACS, a detailed annual survey of 1% of the U.S. population on economic, social, and housing issues, has traditionally come in three versions – a one-year estimate, a three-year estimate, and a five-year estimate. The Economy League, like many organizations, viewed the three-year estimate as the “Goldilocks” of Census data – capturing enough of recent trends while providing a reasonable margin of error.
With the discontinuation of the three-year estimate in 2015 due to budget cuts, we were left to assess the benefits and drawbacks of switching to one-year or five-year samples for several World Class indicators. The one-year sample provides the most current information, but the smaller sample size results in a higher margin of error. The five-year sample is more reliable with the lowest margin of error, but can obscure recent trends by covering a longer time horizon. For an indicator such as the poverty rate in the Greater Philadelphia region, small changes in sample error can result in over- or under-counting by thousands of individuals. While there are drawbacks to both surveys, the Economy League will be using the 1-year estimate for its indicators moving forward to better capture changes on an annual basis.
Budget cuts at the Census have also resulted in long gaps between collection of special data sets. The Census Transportation Planning Product (CTPP), based on data collected between 2006 and 2010, provides invaluable data on commuting patterns. The Index indicator on transportation choice – highlighting the share of jobs situated in communities with multimodal access – uses these data, and the Economy League has also used the data to support specific studies like the King of Prussia rail analysis. The next CTPP dataset will likely not be available until 2019, meaning our region could be making infrastructure decisions in the interim based on commuting patterns from a decade ago.
The Way Forward
The World Class Index relies upon a variety of data sources to track Greater Philadelphia’s progress. Each year, part of the work in updating the Index is not only analyzing trends in the indicators, but also in monitoring changes in the source data and methodologies which continue to evolve. Changes in data availability and collection are not likely to slow down as budgets tighten, new measurement approaches are tried, and even as organizations lose momentum in tracking certain data over time. At the Economy League, we know that we will need to be flexible and ready to adjust the ways we analyze issues, finding the best options for monitoring and better understanding our region.