Organizational theory

I’ve been very excited to learn about all sorts of principles from the field of organizational theory. What is most engaging is how they so strongly relate to many principles within the field of software architecture. Conway’s law establishes this principle, but I think we can take it further.

Conway’s law states that the software developed by an organization mirrors the team and communication structures of that organization. The reverse conway maneuver applies the inverse of this law: structure your teams to reflect the software you intend to build.

This law and maneuver have been established and are well known. Just about any book on microservices or software architecture makes reference to the law in their chapter one and the content of the Team Topologies book is almost entirely dedicated to understanding and leveraging the law.

One aspect of this application that I have not heard is how we can use this law to learn about and design systems that have nothing to do with software. Software development and architecture provides us a test bed for novel communication and processing patterns that we can iterate on rapidly and experiment at scale. The reverse Conway maneuver is used to design software organizations, but all other organizations could benefit from similar principles.

We could leverage the learnings from building software systems to build other systems, like designing administration of social services, government operations, or schools. All of these and others are examples of large scale organizations, possibly distributed or monolithic, that iterate extremely slowly. You can’t just delete people or repurpose them like you can with Terraform modules and cloud infrastructure. The consequences of poor design is much more severe, but many of the organizational and communication principles of designing these systems are the same as in software. Teams have to communicate, either synchronously or asynchronously, and organizations have bottlenecks.

I hope to provide more details and specifics with case studies over the coming years.

Small-scale Machine Learning

So much of the machine learning dialog focuses on a scale unattainable to most companies and engineers. The research literature is too far-fetched or sandboxed to be applicable, and the engineering and design principles only make sense at Google or Facebook scale.

In the past two years I have learned much about designing and deploying machine learning systems on small teams at relatively small companies. The practices we use are inspired by, but diverge in many key ways from most of the literature. Fundamentally, small companies do not have the resources to maintain infrastructure teams and data science teams. The data scientists have to do it all, or nothing gets done.

I would like to start painting the picture of what machine learning looks like when you’re not multiple teams and unlimited resources. What is the best order of infrastructure to build? What kind of projects should a new team start with? And most importantly, how should a team grow? The Team Topologies book provides a great resource for software in general, but their are many aspects of machine learning teams that are unique.

Some key takeaways that I would love to read a blog about: - How do you transition a DS team from a complicated subsystem team to an enabling team supporting embedded data scientists? At what scales do these make the most sense? - Experiences of why solely embedded or solely complicated subsystem teams don’t work. - My experiences with serverless machine learning.