00:00:12

Hello everyone. My name is David blizzard. I am giving this talk today from one of the classrooms at the university of New Hampshire, because I figured this was a, an appropriate venue to use given that my work is in higher education in technology supporting higher education and really everything that I do, everything I work on is really in support of what happens in this kind of a room. It is about the actual learning that goes on. I am based at the university of New Hampshire. Um, I have been there for a number of years and most of the work I'm going to talk about today focuses on things that were specific to UNH. UNH is the lead institution for the university system of New Hampshire. And it's a research one institution about 15,000 students, but I'm now part of an it organization that has been consolidated across all of U S and H. So Keene state college, Plymouth, state university, and granite state college. The other schools that are part of the university system in New Hampshire are also areas that I now do work and have responsibility.

00:01:14

This is a chart of the interest in the term dev ops as measured by Google search over time. The term dev ops was coined in October of 2009. So that's where the chart starts. And it goes up to a peak of a hundred percent, the highest level of interest the Google ever measured. And you can see that August, September, 2018 is about the 50% mark. That is where, um, over the last two years, we've had a doubling of interest in dev ops, at least as measured by Google searches. I'm showing you this to have you compare it to this next chart. This is a month by month. Number of items that are posted on the website for EDUCAUSE about dev ops EDU cause is the premier organization for it in higher education. Uh, all about the uses of technology, both in running the organization, as well as technology in the classroom and in the learning environment while this trend has gone up over time, uh, it is certainly not going up at the same rate that it is across the broader technology spectrum as measured by Google.

00:02:22

Um, you know, we hit a high of five items one month that was, you know, huge. Um, and I got to honestly say that many of these dots here are me and things that I've done presentations I've given talks I've been involved with because I've been involved in trying to promote dev ops ideas into the higher education space, but it really hasn't connected very well. And I think one of the reasons for that is the way that dev ops is often marketing. If I think back to 2016, when Alliance and colleagues first went to a dev ops event down in Boston, we were hearing stories about how a 70 worldwide developer team all got together and on the same page and used a, you know, continuous integration and pulled all their work together in that way. And I'm saying, you know, when I have two people working on a project that's huge, um, you know, we've got to get out the door first.

00:03:15

If you don't release your code tomorrow, your competitor's going to do it instead, and you've got to beat them to market. So you better, not just at least once a day, but twice a day or five times a day or 10 times a day. And I'm just like, this doesn't make sense to me, right. Is there competition in higher education? Sure. But it's not the same thing, right? This doesn't make sense. Right. We're also not, you know, a cloud native organizational structure by default. So the things will shorten the shifting in that way. What is going on in higher education now is a lot of transformation and other ways, and this is another snapshot from EDUCAUSE of the term digital transformation, digital transformation. I was going to make a chart of that one too. But when I actually saw the numbers here, I'm like, I'm not going to waste my time because of the fact that there are over 2000 presentations and over 1500 articles and so on and so forth makes the point pretty easily that, yeah, there's a lot more interest and a lot more discussion around digital transformation at what that means.

00:04:15

So what is digital transformation? Well, it's defined as a series of deep and coordinated cultural workforce and technology shifts and these lead to new educational opportunities, operating models, transforming institutions, operations, teacher, direction, and value. There's a lot of things here that connect with dev ops right away. We're talking about culture. We're talking about technology, we're talking about value delivered by the organization and how we do things. So there are definitely some similarities and connections. And for all of you who are a ops technology companies, dev ops tool providers, dev ops service providers, if you want to reach higher education customers, you need to talk to us in our language. And this actually works much better than just sort of talking about dev ops and go fast and go quickly and get stuff out the door and so on and so forth. There are plenty of challenges in higher education.

00:05:14

Now, as I was putting this presentation together, this was an email that came out from our chancellor for the university system in New Hampshire, went out to all employees and it was just so spot on the things I wanted to relay that it just figured I would copy paste and use this. Here. We are feeling the effects of a baby bust that happened about 20 years ago, a decline in the population in the United States and the number of births. So there are fewer 18 year olds coming into traditional college environments this year than there were last year. And there will be fewer left next year. And if you happen to be in the Northeast of the United States, like we are, it's even worse around here. Um, there are changes in terms of what is the value of higher education and that being discussed, how do we deliver education?

00:06:02

Do we have to be in a classroom like this? Or can we be online? And if we're going to be online, what does that mean? And should we be teaching in the same ways in the same modalities and the same topics, or should we be focused more on making people ready for business and marketplace needs for workforce competition is going up and the funding is not, um, New Hampshire ranks 50th in state funding to higher education. And if we doubled the amount of money that the New Hampshire state government gives to higher education, we would still be 58. So that isn't likely to change anything anytime soon. And then you put on top of this COVID-19 we had to close down in the spring. Like everybody else, a lot of students went home when they went home, they worked on campus using the dining halls and the residence halls and other services and so on and so forth. So it was a lot of money that had to be paid back. Um, so really big challenges that UNH, the other UNH schools and higher education are facing as a broad general trend and even more so now because of the pandemic.

00:07:10

So with that kind of background laid out, I want to talk to you about our transformation and our work related to dev ops and how we've gone from where we were to where we are today. I'm gonna be focusing on work that was done by our team, responsible for our core UNH websites and web applications. And this is what our box is. Our environment looked like we had three actual physical server boxes that everything around databases, website, applications, load balancing. Um, if we had dev and test environments, they lived on there. A lot of times we didn't, um, it hosted static websites that hosted suffering, Drupal six stuff and triple seven, but a little word press thrown in there because Hey, why not somebody wanted it? So sure we ran it. Um, cold fusion was what we did development in. Despite the fact that cold fusion often ran out of control and broke everything.

00:08:00

And because it was all shared infrastructure, everything went down. Um, and then we started working on some maintenance improvements of this. You know, we had one awful deployment that was a Sunday morning death March. That was supposed to be like an hour long, that turned into five hours and failed and had to be rolled back and so on and so forth. And this led to us focusing in on one key thing, more than anything else. And that was just pure uptime. We set up internal monitors and external monitors and just sort of religiously every month looking at our uptime and doing whatever we could do to make that number go up.

00:08:40

It took a while from there, fast forward to 2014, this is the new improved version that we got to, which was based on virtual machines. We were able to separate out functional environments, separate out technology stacks. So now when that cold fusion ran out of control, it would still crash the cold fusion servers and all the cold fusion based applications will go down. But our websites that were in Drupal are static websites and so on, they were still alive. So that was an improvement. We could also clone out proper dev tests and production environments. I would not have put it in this term at the time, but really what we were doing overall this time and really what we were doing. A lot of it from here on forward was paying down technical debt. I would have talked about getting rid of legacy systems and, and, you know, out of date stuff and unreliable stuff, but really a lot of things have been built because, okay, that was easy.

00:09:32

Let's get it done and move onto the next thing. Right. I'm sure you're going to have a website for your lobster research, cause why not? Is there a strategic reason for it, for the university? I don't know. Nobody was really thinking about things in that way. So we had over 400, some odd websites. We pared that down to about 200, still a lot, but a lot better. And we started standardizing and having structural reasons for things and making sure that when we did something, we did it well so that we didn't have to go back and redo things. In 2016, we added Docker. Um, we really started looking at containers and as Docker was coming on board and, uh, it was in late 2016 that we first got our Drupal infrastructure for our web applications running in Docker. And that was great because now we could take that idea of isolation of individual environments and build that out to the highest degree.

00:10:30

So we wanted to do that. It took us a long time. Um, it took us about a year, really to build up enough knowledge about Docker and Docker swarm, and maybe looking at Kubernetes and maybe not, and back and forth and working with, you know, pipelines and how do we deploy this and how do we manage this and how do we restart things and all this stuff we had to figure out. It took a long time, but now we're in a place where all of our websites run in a containerized environment. And now not just if one individual application framework has a problem, it's now isolated to the individual websites. So if suddenly our business school website gets a bunch of traffic, it might get overrun, it might get overwhelmed, we can do scale-out to help it. Um, but it's not going to affect everybody else because it's isolated. So that's a huge improvement.

00:11:23

Now I've already talked about how long it took us to roll some of these things out and do some of this work. And there's a reason for that. Um, our budget throughout this entire initiative was $0. Well, not entirely, we didn't pay for an external monitoring tool. That's the only thing we actually bought. So we wanted something good, but it had to be cheap, which means it was not going to be fast. It took a lot of time and energy and effort to learn things, work on solely implementing things. But the result of this is that we've really built ourselves in an environment and a set of tools that needs our needs very effectively and very unique. Wait, we brought on a lot of tools. Um, I've already met some of these like get lab and Docker and Docker swarm. Um, Rundeck, we've really rolled out to have a sort of a central place to have all of our scripts and tasks be able to run and be tracked.

00:12:18

And we're standardizing on Python scripting language to have one language that everybody's familiar with, um, you know, centralized logging, doing configuration maps, judgment for the underlying systems that are behind in the database and the behind the Docker environment, starting to introduce some automated testing tools and things like that. So that's worked really well for us. Um, along the way, as we are working with get lab, I got to admit, we just playing got lucky there. Um, we got lucky because get lab matured along with us, it started out as a pretty way to search your code. Well, by the time we got to the point of really being serious about looking at pipelines and looking at how you would do, um, continuous delivery, we discovered the get lab had added this CIC D functionality. So even though we'd started looking at Jenkins, it really wasn't necessary for us.

00:13:11

And we've done an awful lot of stuff within get lab and get lab ICD. So that was advantageous. So now we've got a much more robust environment. These are stiffly developers, the biggest tools in terms of how our developers work and some of the things they use for their natal and then environments for their testing, making sure everything is ready before it goes through the pipelines or as part of going into the pipelines. And then we've got a pretty robust set of environments for doing testing and rollout of code. Testing part is still under development, but it's coming along. And this way we can have a way that we can get stuff out the door. Now I said, right back the beginning of that, this whole idea of release 17 times a day. It doesn't make sense for us, but we now can release 17 times a day. If we wanted to, we don't need to, it's not the reality of our situation, but if we had to, we can deploy code out the door very quickly. If there were a critical security fix that needed to be done to Drupal, we could have it out to all of the Drupal based websites we have in under half an hour. And that's fantastic. And that's just because we've got a robust, reliable infrastructure in place.

00:14:24

There was another part of our journey to talk about, and that is about data. This is the way I described our central data environment recently in a meeting, somewhere in the woods of New Hampshire, hidden away, there is a castle. It is surrounded by a moat that is where our core institutional data lives. And by the way, yes, there are dragons as well, guarding best. Um, this is not really true, right? But if you look at where our data does live in our actual data center, which is, this is the outdoors of it are the outer door of it. This is actually a old food storage warehouse. So a big giant cinder block of a building, um, way this is set up is that our data does live very much in that very much isolated, molded, protected zone that you can't get to, which means you can't really do much with them.

00:15:24

And that's a challenge because we have a lot of court data that we work with. We have our students, all their basic contact information, their status, their gray, um, their, their history, their majors, their, um, you know, age, all, everything you can mend. Imagine the demographics, their socioeconomic status, are they veterans or not all sorts of information about students? We have information about our faculty members. We have the courses that we're offering. And of course we have to actually put faculty and students together into a course. So we have all the enrollment information. And if you are enrolled in a course, you get graded. So we have grades that are recorded. And then there are financial transactions and financial details behind all of this as well. So this is the kind of data we have that we work with cross the university system. Um, there are some other things, there are medical records in certain places and there's specialized data for housing or dining and so on.

00:16:25

But really this is kind of the core institutional data within the kinds of tools and technologies and systems that I and others, my colleagues work with. If you look at some of them like learning management systems, to provide an environment for faculty to share materials electronically for students to review those materials, to submit homework, to get feedback, to manage the grades. And so on that needs student information, faculty information, course, information enrollment information. At the end of the term, we do evaluations on all our courses, which needs student information and faculty information, and course information and enrollment information. Um, we have web directories, which needs the student information and the faculty staff information. We have department websites want to highlight their faculty and what they do, which means we need the faculty information and their course information. We have an exam scanning service, which if you do one of those, you know, automatically scan, bubble sheet kinds of exams, we need the student information, right, again and again, we keep coming back to the same core data elements that we need over and over.

00:17:29

And over the way we used to do things is that to get to that data inside that castle surrounded by the moat guardian by the dragons is for each of these things. We would have a custom pathway, built a custom feed of data to go in and get those exact elements we need for that one thing and bring it back out. That was a lot of work and a lot of duplication, one very specific example. We wanted some data, basic faculty, staff data for, from HR system. As we were working on building out our course, our college and school websites, just basic who you are, what department you're in, um, you know, your name, your address, your office, location, your phone number, your email address. Um, some basic things like about it took us eight months to get that because it had to be custom built for that one function.

00:18:24

What we've done is set up what we call our data hub, where we now don't feed data to each application individually. We pull that core institutional data into an environment that is available to our web applications so that we can get to it. We use Maria DB, there's the backend for it. And in this data hub environment, we have all this stuff sitting so that when we have a new application come along, if it needs course data and faculty data and student data and so on, we already have it. Now we do still have processes in place. There are still approvals that go into effect to make sure that we're doing the right thing. We're not selling our data to somebody we're not giving it away. We're not exposing things we shouldn't. But as an example, another time we needed to get data that was coming out of that data hub environment.

00:19:11

It took us two weeks to go through those approvals processes. And then we can get on moving with things much more effective, much more responsive, gets people what they need much, much better, way, much more quickly. And really we've taken this idea of working with data from a central place and built that as a core principle for our environments. We have one infrastru and Drupal now for all of the different websites, each of the individual schools within UNH. So there are 15 unique websites, the Marine school, the business, the school, the engineering school, health and human services, liberal arts and so on and so forth. All of those pulling data from central sources, HR data out of our HR environment, research information about faculty research interests and their personal bio or CV. And so on. Come into there, what's going on on campus from our central calendar system, what our degree programs are and the requirements for them.

00:20:10

So that that information is not duplicated. It is brought over as the one official copy of that information. Never go stale, never goes out of date and also news items that are built in another thing. All of this goes into this one infrastructure. We maintain one infrastructure. We use that infrastructure over and over again. We suddenly come along and buy a new college. Somehow we could roll out web port site for that college in about a day's work. Um, getting all the data fit into our environment. We have it there, roll out the website and it would be pretty functional with their programs and their staff and so on and so forth also because we've got one infrastructure that's shared. Anytime we do fixes enhancements that get rolled out across all these sites. This has also led to a much better user experience for our students on our staff.

00:21:06

This is what our websites looked like before pulling all of the headers from all the different websites you will see sometimes about at the beginning, sometimes about 16. And, um, we have apply how to apply. Sometimes we do. Sometimes we don't. Sometimes we get things about giving some, since we don't, sometimes we call it people. Sometimes you call it staff sometimes called faculty. It just all over the place. Now with one infrastructure, we have one consistent look and feel. Here is liberal arts. Here is Marine school. Marine sciences here is health and human services. They have the same structure, the same look and feel. You think you're in one organization that actually knows what it's doing, because they all have the same information presented in the same way. It's much better service. It's accurate. It's up-to-date. If there's a change to the requirements in a degree program, as soon as that's rolled out, it goes onto the website immediately. If somebody's office moves and it gets updated in the central sources, it's on the website immediately. And while we are not releasing 17 times a day, we are releasing much, much more frequently. We used to where you used to be. We'd work on a website and then maybe not get back to it for months and months. Now, over the time we've had this in place for the last couple of years, we've been releasing about every 10 days new functionality or upgrades to these websites.

00:22:31

So what's next. Um, as I was putting this presentation together, I was looking back through some old things. I found this presentation from that event. We went through down in Boston in May, 2016. This was actually given by Jane Raul from the dev ops Institute. And it was this slide about dev ops is a journey. It starts with a single step. It has no finish line. Um, and I just really, really connected back with me, um, when I was doing this. And that is exactly where we are. We're not done. We never will be done. We just want to keep improving from here. So what are some ideas? Are we thinking about doing things new and different, going forward to continue that digital transformation, this is another thing to measure Cox. It's a list of all the possible types of services you might have across higher education.

00:23:16

If I look across all of these across the institutions, the ones I've highlighted here are places we do development. Everything I've been talking about so far is in that very last item down there, web services, right? That was our websites. We do, we do development there. We do research related development. We do development around integration to connect tools together. We do development around reporting, but across a lot of these things we don't develop. We don't build our own learning management systems. We don't build our alumni systems. We don't build our own facility systems. We don't build our own HR systems or finance systems. We buy them and use them. And so we have this sea of applications that we deal with. And my current role is as a director of application administration across U S and H. And so I'm thinking about how do I take these dev ops ideas and apply them across this kind of a space, just because dev ops has dev in the word, um, implies development doesn't mean that the ideas, the idea of having shared responsibility for an entire system and service, being able to measure things, to use those measurements, to drive improvement, to automate things, to do more reliable, more effective release and delivery of services.

00:24:32

All of this is things still make sense. If I have an application, it requires our database services, our system administration services, our identity services or security services to all work together. And so a lot of this makes still a connection and makes sense in terms of talking about how we should run and manage things. Um, there are also specific pieces of things. And I'm going to give you three really quick examples here of things that we're just starting to experiment with automated testing while we're not developing our learning management system. Why can't we test our learning management system on a regular basis when the developer releases new updates, and this is a SAS product. So they release stuff all the time. How do we know that it still works? Especially if we make configuration changes, we have some things where there's a certain feature. There's no real way to turn it off.

00:25:24

So we use some JavaScript or some CSS to hide that button away. So people can't use that feature. How do we know that's the case? How do we really know it's up and running and functional? Well, if we can mutually right testing, functional testing using something like B hot or selenium, we can get something like this. Here is a proof of concept for validating logins for our learning management system on the left, we have a situation where it goes through and it does a login test that fails or that, sorry, that passes. It does a login test with a bad password that fails just like it should great that to pass. And then we try to do something where we should hit a certain webpage and we redirect, and we didn't by comparison on the right. You can see a successful test, again, very simple, but this is the kind of thing that we're starting to build out for some of our core applications so that we don't have to validate them ourselves by hand over and over again, we can set this up to run when we need it, or even better.

00:26:16

Let's run it every 15 minutes and just know that everything works, applying the concept of doing things in code. Instead of doing it by hand, a lot of web applications, you go in, you click the button to add something, you type it. You sit say, if you go next again, again, repeat typo, fixed change. You're doing all this stuff and there's no permanence to it, right? It's kind of boring. If you've got to do a lot of just manual entry stuff. Well, we had an opportunity with our data hosting or our media hosting environment, where we set something up new for undergrad research conference in the spring when they couldn't do it in person, we gave them this as an online platform to share all the student work worked really well. We were able the build out that configuration in code, there were API APIs available. We could basically build this, upload it into a test environment, get the department to look at things. Okay. They had some changes between the source files. We upload again, looks good. All right, exact same thing gets loaded in production, much more quickly. And we've got it as a permanent artifact. We can reuse over and over again, applications that allow this kind of modality of working, instead of just, everything must be done through button collects much better something we definitely want to encourage.

00:27:34

And also taking the idea of microservices, little tiny add on functionality. One simple example. Faculty members want a course in our learning management system to do some things so they can try something out. We call it a sandbox courses. We don't normally want faculty to be able to create their own courses that might lead to problems. So instead, if we had a simple button that they could click on to request the sandbox course, we captured the information. It does the work for them. That today happens by a person intervening. They submit the form. Somebody actually goes in and does the work let's automate it. Let's speed up delivery, simple little services dealing with students who have incomplete center core. So at the end of the semester, for whatever reason, they can't finish the coursework. They're allowed to continue on beyond that. But the course closes down the fact that the number has to ask, we open it back up. Why can't we have a microservice that shows the fact that they listed their courses. Once you pick a course, it shows you a list of the students. You pick a student, you submit a request for an incomplete reopening of the course of that student. It just goes, and doesn't little microservices to give better functionality,

00:28:41

To close things out. This is my definition of digital transformation. Taking away the rough edges, taking away the technology problems and weaknesses taking away the over-engineered over-designed solution. That has been the way we've always done things. And the silos that separate keeping us from supporting or really trying to do our real jobs or support higher education and to support learning. If we do that well, that is what we want to accomplish. And to me, that sounds an awful lot like dev ops. Thank you very much. I hope you enjoy the presentation. If you'd like to talk more about it, please reach out good day.