Interview with Richard Matzner

0:00 Okay, now we're actually recording something, and it's the 20th of March at 10.30 in the morning, and I'm speaking with Professor Richard Mattson. Okay, so is the format questions or what? Not necessarily. I mean, I'm interested in the background to the Grand Challenge and how and why they're going to regionally. Let's see. In 1991, I suppose, it must have been, there was a meeting at the American Physical Society. It was the 75th anniversary of the Einstein field equations. I believe the APS meeting was in Washington, but I would have to check to be sure. but there was a session organized by Cliff Will on gravitation because it was the 75th anniversary of the publication of the equations. And I was there, and I spoke with some people, including Ed Seidel and Richard Isaacson, about the binary black hole problem. Ed was working at the University of Illinois on that problem, and I was doing three-dimensional cosmology. Ed's approach was then two-dimensional, axisymmetric, and so in some sense we were approaching the 3D black hole problem from different directions. I had never actually physics of that type. Anyway, that conversation with him and with Richard Isaacson led to the idea of trying to organize an effort to do the real binary black hole problem. And after a lot of communication back and forth that led to a proposal in 1992, I suppose, to the first Grand Challenge effort, and then in 1990, which we did not win. In 1993, a slightly improved, well, improved because it had a lot more attention paid to the computational questions proposal was funded,

2:30 and that became the Grand Challenge, which started in the fall of 1993, and which has eight universities as principal research institutions plus two other associate universities. universities there, St. Louis, Washington University in St. Louis, Wymosun is an associate and University of South Africa. And then we also now have an external associate group, which is the Potsdam group. After N. Seidel moved to Potsdam, we worked on an agreement whereby they are foreign associates, I guess, is the way. So it's a pretty big effort. And it took a while to get organized prior to the proposals and took a while to get organized in the sense of deciding what the right physics was to carry out after we got the program started and is now producing results at a good clip. The end of the grant, the anniversary end of the grant is next September, but we have a one-year extension. I can moderately confidently say that we'll achieve most of the proposed results, but probably not by next September. But I'm pretty confident we will do some in the following year. The one-year extension is an unfunded extension. But do you feel that the progress up to date has been such that it would be possible to continue on in the next year without being... Well, yeah, that is a problem. We haven't yet fully got funding for the post-docs who are the central people who are going to work on. I would be very happy if we had a year of funded time to go at this point rather than only half a year.

5:00 Our timetable has definitely slipped by order of a year. we found out we've discovered a lot of stuff about the physics of the problem and about the computation computational problems of the problem I can't still have a couple of difficulties that aren't totally under control but look as if they will be brought under control in the next few months and with those a matter of putting pieces together. Statements like that are very dangerous to make, but it really does look like it's a matter of putting the pieces together in order to do an orbit, several orbits, say, and a merger of a black hole, and to extract radiation from it at the same time. Now, at the moment, these things take a lot of computer time. computers are getting better, but they're not getting better as fast as they were projected back in 1993 to get better. So in terms of making huge numbers of simulations, we may be inhibited just in terms of computer resources. You mentioned the original proposal, which was then modified successfully earlier. How different were the two proposals? Was it the same group of people that was involved in the proposal? It wasn't exactly the same group of people. So the original proposal had six institutions, Texas, Illinois, Chapel Hill, Northwestern. I'll have to check my list. Let's see. Well, I'm not thinking too clearly today. I'm sorry. Excuse me a minute. But if we can come up with a list of the institutions, which I can do if I get on the web, then I can tell you the ones who weren't in the original. Sorry.

7:30 Okay, Texas, Illinois, Chapel Hill, Cornell, Pittsburgh, and Northwestern were in the original list. The current list includes those plus Syracuse plus Penn State. Penn State was added because one of the people in the second grant, by the time we got to the second point, Pablo Laguna had gotten a faculty position at Penn State, and he is one of the substantial contributors to this effort, so he became a co-understander. Syracuse was added because Syracuse is the location of the Northeast Parallel Architecture Center, computational science institute which concentrates and in fact was was the center of development of what's called high-performance Fortran which is a parallel a parallel Fortran Fortran for parallel processors and so one of the results of this was that we had a lot of development on HPF originally high performance Fortran. And this is an example of something that didn't turn out as we expected, because what happened was we discovered deficiencies in the definition of HPF that restricted the kind of adaptive adaptivity that we saw we'd have to write for the codes. So, in other words, HPF just turned out not to be as useful a language as we thought it would be. And as a matter of fact, HPF seems to have peaked. It appears that HPF is no longer considered a viable extension of Fortran 90, and it's losing ground. It's becoming a backwater of programming. So we were one of the first to discover that it didn't completely address all the problems.

10:00 So the second proposal included Syracuse strongly, and it included the computer science department here at Texas, and it included the computer science department at Illinois. So the second proposal was strengthened principally in that we got computer scientists who understood parallel systems scientists who understood the solution of equations, computational solution of equations. The previous version, I guess you could have said, the physicists thought they were going to do it all. I mean, there were people from physics, astronomy, and math departments, but they were all relatives, essentially. And the new version, a very basic thing that happened We got all this input on Fortran, high-performance Fortran, but we also got DAG-HDH, which is what you were talking to Jim Brown about, which does work, does let us write single processor codes, which then are spread over multiple processors. We currently have jobs running using it. DAG H also handles adaptivity, and we have people working on getting adaptivity into our codes. Our codes are very big, and that aspect of DAG has not been pursued as vigorously yet, but we're working on it very hard right now in order to get adaptive refinement into the codes also. The number of variables in these codes is on the order of 100 or more. and we're currently running something like 128 cubed runs so there's i believe that's 2 million times 100 that's 200 million variables each of which occupies 80 bytes so basic code size is on the order of 1.6 or 2 gigabytes just to execute these codes. And parallel machines around that are accessible to us don't have tremendously large memory compared to that.

12:30 There's a parallel machine here in a T3E that has 7 gigabytes if you've got it all. There is a 28-gigabyte machine at Illinois that the NSF has provided. But we would like to have higher resolutions and higher speeds. And so you can see that we're pushing the NSF-available computers. It's nothing like what we'd be good if we were running as a DOE project, but we haven't managed to acquire that Aura yet, if we will ever. So there's a certain problem of being limited by the data, but you can't have it? It's not just the time, but the size of the computers. It's a substantial problem. It has never been an essential problem, but it has always been an annoying problem. We can't do as many runs to evaluate as many parameters as we would like. We've made progress nonetheless and probably would have made, conceivably could have made, you know, 20% faster progress if we'd had access to larger, faster machines, which could be, you know, and then we wouldn't be a year behind. I don't know how seriously to take that because there's a lot of understanding of the behavior of the codes and of the system that has to go into making progress. So apparently the kind of multidisciplinary aspect that's strengthening the computer side was something that the NSF was... Well, yeah, they were very... It was, the project was run by the computation information, computation, no, information science engineering, it was called CISE program at DSF and they specifically are CS computations, computer science oriented. And so the projects were, since this was funded as a Grand Challenge, the purpose of the Grand Challenge was to show that we could run large parallel codes and to show that we could collaborate.

15:00 So it was supposed to be a large multi-institutional effort. These Grand Challenges were not supposed to be single institution efforts, and ours is certainly not. Was that something that you sort of applied under originally? Well, we applied to get funding for a large number, say an additional half dozen post-docs and associated travel money, et cetera, to do this work, and we wrote a couple of proposals which were very, they were real proposals, but they didn't have any program to submit to, and the amount of money was more than could be obtained from gravitation. So when the NSF announced these computational grand challenges, this seemed a perfect fit, and we then applied for it. As I said, the physicists would have been happy without all the computational baggage. They would have been wrong also, because we could not have accomplished what we did without the computer science. But physicists are a, what's the word, self-commoning. So in the event, was it an easy merger between the two disciplines? Oh, actually the computer science side of it worked extremely well because computer scientists are familiar with the idea of large projects and compatibility among software and making all that work. And as soon as it worked to any extent, the physicists took it up because we appreciated the fact that it did work. Now, there are certainly aspects of DAG that annoy people, and we have other products, one of which is a sort of automatic script script that will take a single processor Fortran 90 program and automatically insert it into DAG. That isn't specifically part of DAG, but it's a very useful product. And we also have scripts that will take a high-level description of the problem and write

17:30 Fortran or C Those things are secondary kinds of computational science things. The latter one was developed by a physicist here in the group, Robert Marston, one of the postdocs. So there is a bit of a conflict in the CS in that we thought we were going to use HPF, high-performance Fortran. and it was a bit of a battle with the people who started out thinking that once we realized we had to shift for more because they had invested a lot of effort in it and wanted to continue. But that battle lasted only a few months. There were more severe battles about the physical science, physics science approaches that lasted longer, but which have all converged to a single effort at the moment. So, as a general rule, there were longer-lasting debates over what direction to take in the physics. Yeah. These debates were whether we should use, for instance, rectangular coordinates or some sort of coordinates adapted to the surfaces of black holes, whether we should use what are called a singularity avoiding coordinate system or whether we should use black hole excision techniques. The singularity avoiding coordinate system essentially does model the inside of the black hole, but you don't have instant singularity in the black hole. It takes a while for any system, depending on how you set the data. You can set the data so it takes a while for the system to evolve to hit the singularity. So you can make one description inside the hole and out, as long as you make the forward step in time very small inside. This, however, means that something that's called a simultaneity surface has pieces of

20:00 it that are late, pieces of it that are early, and you get very strong gradients where you somehow have to match this stuff. So it leads to actually coordinate singularities that no one has managed to control. Alternately, you can say we just will never see what's inside the horizon, we're not interested with what's inside the horizon. And we excise that and use a method of differencing that respects that causality. So we do not allow things to propagate out of the black hole. It's called causal differencing with excision and it works pretty well. And that's the method that we currently are working with that has no apriority reason why you can't evolve a black hole forever. We have evolved black holes forever, actually, with it. In the case where you're actually doing a Cauchy evolution and where you can set boundary conditions which amount to giving fixed exact boundary conditions, then if the boundary conditions are moderately close in and the resolution is moderately fine, we then can evolve a black hole with an excised interior forever. Those conditions still are required. We haven't understood yet whether we have a resolution problem or a domain problem. When we try to make it a little bit bigger domain, we get instabilities, which shorten the lifetime to some number of 200 animals that are infinite. But, anyway, the method, the excision method works, and it's the one we're currently using. Then there was a big, one of the big results, a theoretical result of this, was the renewed interest in the hyperbolic schemes for describing the Einstein system. and York and Chouquet-Brois and a number of people developed some very interesting schemes. These things effectively deal with higher derivatives of the Einstein equations. Just take a time derivative of the standard Einstein equation.

22:30 There's an equation which is the time derivative of the extrinsic curvature. I mean, the way we look at it in the Cauchy case, it's a time derivative of G equals roughly the expensive curvature, and a time derivative of the expensive curvature equals a bunch of stuff, including spatial derivatives of G. Well, if you take another time derivative of those two, you get, well, if you take a time derivative of a second time derivative of K and write that you discover the system turns into what looks like a pretty clean wave equation for k plus some terms which are just straightforward take a step forward in time for g so g dot equals k that doesn't have any wave character or anything like that that is one of the equations then you have these other equations, which look like a non-linear wave equation. You also have some gauge conditions on the lapse and the shift, which introduce weird terms, but can be gotten rid of, for instance, if you assume a harmonic time slicing. So essentially, this is a higher order equation for it. That represents the Einstein equations. It has the same initial value problem so you have to solve the hematonic momentum constraint. Then you have to solve one more constraint which essentially is the k dot equation expressed at the first time. So there's a third constraint. And then you can show that you can freely evolve this wave equation and maintain those three constraints analytically. So, this system and similar systems, York and his friends, his collaborators, have come up with a large variety of different approaches. Anyway, those things were looked at by the Cornell people. People at Cornell were interested in them and spent a lot of time coding this up. They called it the Empire Code because Cornell is in New York and these are the New York variables.

25:00 and they spent a lot of time on it in the end we decided we had to concentrate on one code and we had a better performance and unless in the sense of unanswered theoretical questions using a standard our ADM code, which is just G dot equals K and K dot equals something. That's the code that we're currently using. That was the longest and most seriously debated scientific difference. We also have had this ongoing scientific difference about how to handle outer boundary conditions. As a result, we have got some extremely good outer boundary conditions for the problem. One method matches the solutions to a perturbative solution, uses the perturbative solution to feedback in on the data to keep the system running properly and extracts the waveform from the perturbative form, which you can then imagine being propagated out to infinity because the perturbation equation is easy. The other method is to match directly to a characteristic solution, It's a solution that doesn't have spatial, space-like time-conscious slices, but it has null time-conscious slices. Time-conscious slices are null cones, distorted cones, going out to infinity. And that method can evolve all the way to infinity in a finite amount of computational effort. So it's even more efficient than a perturbative form. And if you can achieve this match, you then have solved the Einstein equations fully, not only completely from the center to infinity, and you can get the waves out of infinity and strengthen the wave. So we currently have that working. That is, the codes are put together, and we can run them for a little while under test cases, This is one of the things that we have to get under control.

27:30 The other thing is the fact that the Koshy Code will not run a single black hole forever. In some situations it will, in some situations it won't. We have to push the stability better to the Koshy Code. Those are the two remaining hang-ups in terms of really completing the production of the system that will produce the answer. It's a question of getting enough computer time to produce the answers and understanding any other nasty things that crawl out of the code that we haven't had a chance to see yet. So the main two main problems would be getting the characteristic, matching to work? For a long time, yeah, stabling and getting the interior code to destable. It's possible, I don't understand why this interior code behaves as it does because we just haven't had enough opportunities to test it. But it's possible, for instance, that we could bring the matching region in close enough so that the interior code would be stable. As long as the match and the exterior code were stable, that would be fine. But I would prefer if we could get the interior code on large domains stable also that we would have just one less source of instability under control. We have one more. Just trying to understand the relationship between the different codes, you mentioned that the Cornell people had developed a code. That one is currently essentially mothballed. It's used to test things like DAG, the only version of DAG, and on the parallel version of that code. A lot of the logic in that code was used in the current code. For instance, the causal difference was first developed from the environment and moved to the current ADN code. And a lot of the knowledge from that code, such as the fact that certain terms caused them problems, has been transferred and used this knowledge in some detailed ways. and state lines of equations and the current version of the equation. So by and large, all the codes might make use of codes like Dan Page

30:00 when running out of parallel machine. That would be sort of underlying... Yeah, now of course other people are interested in these codes and we also have a scalar version and we have this scalar meaning single process of version. Sorry, not scale-up, but a single-questions version. And we have this script that will turn such a thing into a DAG code. But other people are interested in the codes and may turn it into a parallel version by some method other than DAG. Nonetheless, all of these codes have been run through DAG and all of them run under DAG. Well, we have the option, I don't know what else to say. It's much easier to develop if you don't run it in DAG because it's faster. You can pursue ideas more simply if you don't have to pay vigorous attention to requirements for parallel variations. Of course, then you try to put it in DAG and you have an uncomfortable two weeks trying to figure out why the heck that doesn't work well. Whereas if you had stuck in DAG all the time, you would have discovered as you went along that you broke something and fixed it like that when you still understood what you had done. So in general, was software compatibility a difficult problem between the different efforts? Oh, yeah. is very, I mean, literally at the level of taking modules and moving them from one code to the other, subretimes or collections of subretimes, and running them under DAG. So you were able, essentially, to actually move code around between them? Oh, yeah. The way we do it is we have a central repository, which is at Syracuse, and you take code out, you can modify it, and then once you're happy with the modifications, you can check it back in. And we require people to demonstrate that it works, and then check it back in. People are pretty good about that. but occasionally you can howl at something no longer we'll compile or doesn't remember.

32:30 We can usually figure out who broke it. I was kind of interested, I suppose, from the sociological view, in the grand challenge in the sense of the previous culture of people involved in relativity, and obviously many of the people here are involved in this, it seems to be very much individuals in small groups, whereas here, of course, you've got sort of like a theoretical big science. How difficult was it adapting to a sort of what seems to be a new culture? Well, I think most people still have that culture. It was difficult, the principal reason being that there were a lot of different approaches when you take to this problem. It wasn't clear when we started that there was a specific clear winner for which way to do it. So it was natural, in fact, to let people for a couple of years develop their approach to their ideas, but eventually we focused on one idea that is difficult. I have had some advice from the NSF that I should more strongly direct money and effort into certain areas. I had that during the course of the program. And I had a lot of resistance from even pretty minor changes in terms of the subject area that people were considering. Toward D, for the past couple of years though, that has canceled out and that has been pretty much zeroed out. There has been some money moved around, and now we're running out of money.

35:00 It hasn't been such a large problem. However, at the beginning, it really was a problem. There was a lot of problems, just people didn't feel that they wanted to work with each other. A lot of resistance brings us to coming to the six-monthly workshops that we hold. because people said it was too much traveling. Those things are extremely valuable. A lot of very, very good science has come out of it. And they have strong personalities, and they like to do that. I'm not excluding myself. They like to do things the way they've always been done, the way they want to do it. So you feel one of the key ways in which the collaboration was sort of mediated was through workshops where most or all of the people were... Oh, yeah. We had a lot of interaction there. We also had many, many phone calls. We still have phone calls twice a week on scientific issues, but we had during a period of time also many organizational phone calls. We had lots of battles about attempting to extend the P.I. list, and that sort of thing. So there was a fair political element to the answer. Now, oh yes, to go back, you were saying that by sort of sometime next year, you hope to be running, actually, for several orbits of the viral black hole and the merger. What plans do you have then to sort of go beyond that?

37:30 And this is a question, because actually at some point we will be producing simulations that are of interest in terms of predicting waveforms. And they obviously need to have a connection with them like a detector, the template design. it appears that black hole mergers could be the first or among the first things detected by LIGO if there are, say, 25-so-MS, plus 25-so-MS mergers. Essentially anywhere in the universe, those things are estimated to be strong enough to be detected by LIGO, and they have a signal in the pass band, etc. It would be extremely useful to LIGO and would be extremely useful to the theoretical problem if we could predict the way it was prior to seeing the signals. Anyway, the idea is that if we can get some merger signals, I anticipate and I intend to work on connecting with the LIGO project in order to turn them into useful templates for the LIGO system. turn them into useful interpretational templates for whatever it is that they see. The biggest problem with black hole mergers as templates is that you don't really expect many cycles of strong nonlinear black hole behavior. If there aren't many cycles, then it's a pretty crude, a pretty crude signal would, I mean The template would hit the signal, and in that case, it's not clear what good the black hole stuff is doing or what good the competition is doing. On the other hand, the detection rate should scale as the cue of the signal-to-noise ratio. The signal-to-noise ratio definitely depends on better templates. So there's some sense in which having accurate tempers would certainly be desirable from the LIGO point of view, too.

40:00 So if, as hopefully occurs, you have sort of a continuing connection to the LIGO project, do you have plans to look for further funding? Well, we currently have proposed funding for the next few years to continue the effort. But that's essential. We have something to discuss with LIGO. Yeah, beyond that, of course, yes, of course. And the funding should be easier because the computations will be better and LIGO presumably will be producing some signals and the combination of those two should make it a very interesting subject. So the new proposal that you mentioned, will that include substantially the same groups that were gone previously? No, there have been, I believe, six individual proposals submitted. They were submitted in groups. Penn State, Pittsburgh, and Texas submitted a group proposal. Because we could identify three different pieces that would come together to continue developing the current KOSHI code and the matching. Matching is being carried, and the characteristic part of it in particular, is being carried out of Pittsburgh. Penn State can contribute initial data solutions and evolution solutions and Texas can do evolution and the join the match between the Koshy and the characteristic so and at the same time Cornell Illinois and Chapel Hill have submitted one which I haven't read but which emphasizes the empire approach. Chapel Hill, Jim York is at Chapel Hill, and he's a great theoretical resource for that. And Cornell is where the code already exists, the empire code, and is maintained. And Illinois, Stu Shapiro, who was in the Cornell group, is closely associated, is at Illinois. So those, that group is also requesting funding.

42:30 These requests are essentially to support post-docs, so the three that are associated with what I want to do would support three post-docs. So the Cornell group and that there plan to pursue the other? Yeah, when I've spoken with them, but they're not excluding the other pro-achiever. And we are, you know, in communication, we'll collaborate and communicate with them anyway. And so I guess what is the kind of the current situation as to who's doing what, since you mentioned, Okay, let me go through the list. What we're doing in Texas is we're running a lot of Cauchy codes, developing the Cauchy code, and also this script interpreter, which will take high-level description and write either DAG or Fortran or C code. That's being worked on here. Illinois is principally concentrating on the outer boundary problem, and that means the boundary condition and the perturbative matching. Chapel Hill at the moment is essentially, well, there's a graduate student who may have gotten his degree by now problem, and Jim York, who's working on the theory aspects. Cornell, Mark Shale and Greg Cook are working on the Koshy Code and on the DAG implementation, and on Horizon Trackers, which are a module that you need. They have crude ones. They're working on one improvement to this, which allows you to find out where the horizon is and where it exists. Syracuse is working on

45:00 the adaptive DAG version of code. Pittsburgh is working on the characteristic code and the match between the Koshi and the characteristic code. Northwestern is essentially just advising at the moment. In fact, now that Sam Finn has gone to Penn State, I guess I don't know when that's going to happen. I guess that's next year, so in some sense, the grant will go to Penn State. And Penn State, they're working on initial value problem and on the Koshy Kamehameha. Washington University, WIMO has been using the DAG in some of the neutron star codes and has been contributing a little bit back this way to advice on how it works. University of South Africa, Washington University is an associate at the University of South Africa. Nigel Bishop has been working on the matching program and also on initial data from multiple black holes. Actually, that's going on in Pittsburgh, too, and in Texas. South Africa is an associate. And then Potsdam is an associate. And this has been mostly, well, they've been running some of these singularity avoiding codes, but they've been using them to develop things like waveform extraction with different boundary conditions at the underboundary. That's a lot of activity. Yeah, it's a very intense project, and that's the reason it's actually making some progress. When we started this, there were no stable black hole codes in any number of dimensions, including one dimension. There were some axisymmetric 2D codes. Now we can run stable black holes forever in a characteristic formulation.

47:30 We can run stable black holes forever with some limits on the domain in the Cauchy formulation. including Kerr, not just Woodshore, and we often, any day now, we'll have the data and do the first step of interior excised two black hole evolutions. That is actually doing the evolution the way we want to do it. There have been other evolutions of two black holes, the axisymmetric ones, for instance, and there's a three-dimensional one that Bern Brookman showed about a year ago, No, no, no, last September. Those two used the Singularity of Voting Boarding Commissions, and that has the effect that you're sort of guaranteed it's not going to run for a very long time. Although they have, the Illinois group, Illinois and PASTA, they have managed to get such Singularity of Voting Codes to run for, say, 100M, which is a long time. But none of that existed five years ago when we started all this, and all of that can arguably be said to be a result of the grand challenge, some of the more great things. You mentioned that the ANASAN was interested in... All right, go ahead. How are you doing? Not too well. I'm going to go home a little while. I'm just going to take me to lunch, so I'm going to be a little bit early. I'll be back early, too. Okay, well, I won't be here. Don't have to rush. You mentioned that the NSF was advising a more sort of focused approach than that. Did this entail something that they wanted, you know, somehow that everybody should sort of work on the same thing or that it should be only directed to certain areas? Well, yes, both of those. They were concerned that we shouldn't have, for instance,

50:00 an empire code and a regular Koshi code. and as a result we did the point is that we did narrow the focus based on their objections they didn't want us to have high performance Fortran and as a result we narrowed they didn't want us to have two approaches to the outer boundary condition one of them is the perturbative the other one is the matching Now, that, as a matter of fact, we still have two approaches, but we tried hard to decide which one would be better to concentrate on, but I believe we made the right decision because the perturbative matching works, is not as accurate or as elegant as the other matching. On the other hand, the perturbative method has limitations in terms, in particular, of the strength of the region, the strength of the field in the region in the client. And given the computational restrictions, we essentially need the matching method because it can be applied to a stronger region. It's exactly solving the answer to the question, so you don't have any eight-priority restriction on how far away you have to be. So anyway, that was the remaining one. But there was problems because people wanted to do... There were continuing battles because people wanted to work on things that were not black holes, for instance, put matter in the equations, et cetera, which are very interesting areas in which our developments that are being developed for the subject by other people right now. So, the effort was felt, and may still be felt, to have been closely not focused on Black Oak Island. I would say that there may be some marginal truth to that, but it's a little difficult to see how it could be much closer focused on it was, given the time scale and given that we didn't really know how to do it when we started. So in some cases then where the NSF was anxious for you

52:30 to kind of choose one approach over another black empire versus Indian approach, you sort of at some point decided to go with one another, but you were saying, say, in the out-of-boundary condition, you actually found it useful. In the out-of-boundary condition, we had some real, deadlines set to make decisions, et cetera, but what we found was that the problems that were being developed, becoming apparent in the development, strongly suggested that we should keep both of them going. So there's a real reason why we need both. We need the perturbative one because we needed the boundary condition. And we need the characteristic need a strong field extraction. Interesting. So then in the case of... In a case where you decide to, as you say, to choose between one of the same mothball and one of the cold landing impact boat, Does that simply involve sort of ending the part of the funding that promotes that work in a particular place? Well, no, it just meant that people who were funded, we didn't, we moved a little bit of funding from one of us to the other. No, it just means that people who were working on it worked on Mukeshiko instead, and the The funding didn't change much. So that was basically the approach. So, the... So the political battles that you mentioned weren't so much over funding and that because people kind of had more or less the same size on the law or was it...? slight changes. There was certainly the impression of a battle in the sense that people were very concerned that their slight's being maintained.

55:00 So the political battles were more related to the direction? Which was the science. One thing that somebody mentioned, I forget who was that, another issue they felt that sort of came in on the political side was the question of assigning credit to such a large collaboration that they thought that in GR had been common that there were just a few people in the paper and maybe you just put them in alphabetical order. and that with larger collaborations, people were not used to some standard method of sighting. I've got one of these things. Maybe you've seen them. We already have had it as our letter. Ah, which has 42 authors on it. Well, we decided, there were two aspects to that. One is that we decided that on Alliance publications and that sort of like modern art you know it when you see it we would list everybody and everybody was determined by the PIs as the PIs would say who everybody was so I have ultimate authority for everybody from Texas and Jeff Winokur has ultimate authority for everybody from Pitt. We've had three of such papers so far. There are three Israel letters that have all been essentially accepted. They document the progress so far on the outer boundary, on the moving black hole, and on the infinite life evolution of black holes. Most of the other papers of the Alliance are written by groups of people that are a few, half dozen at most, some collaboration involves a few people. we had one article which was a science article I won't find an example but we actually got a science cover which has that figure and that has only I believe six authors and that was a decision on my part

57:30 was agreed to by the other authors who represent most in the, let's see, I believe it's Texas, Pittsburgh, Illinois, Cornell, and Washington University. I have repeatedly heard that junior researchers, the names are on the eyes essentially, that junior researchers were offended that they were not listed as authors. That's a problem. I'm not sure. I mean, the author list was agreed on before the paper was submitted. But still, it worked out that some people felt they were unfairly excluded. I think that the current situation of having specific alliance papers that have everybody on them followed by, say, on the same subject, a longer paper that's got the specific people on it. It does work to give everybody the credit they deserve. It is certainly a global effort to achieve these things, and I would be... I certainly can defend the presence of every author on that 42 author paper. So what was it a certain thing? The decision to do that was one of those political things that we spent hours on at one of our period. Presumably motivated, to some extent, by the fact that you were saying that in some instances there were the complaints that not enough people were on paper. So earlier, was it more of a policy to have just senior people or PIs on certain papers?

1:00:00 Well, there wasn't a policy. And as I said, that science article was, the authors on that were agreed to by the authors. Certainly it's true that the authors on that paper contributed very substantially to it. So it seemed appropriate. But after that, we did develop the policy. And as I said, the sort of global things that report the results of the alliance are published with everybody's name on it, according to the policy. So you mentioned that most of the money that is asked for, I think you know, the original branch out for postdocs, it goes towards supporting postdocs. Presumably a lot of manpower is necessary for a project like this. Essentially all, I mean, that's effectively all it was for postdocs and I guess students, There must be some students. Definitely our students are supported by them, but we counted 10 post-docs or so that they were supported by them, which is more than the amount of money we got. We, you know, the budgeted amount of money was at its maximum $875,000 a year. A post-doc, once you add in the fringes and benefits, costs $100,000. So 10 post-docs is $1 million. And we got matching, real matching, like supporting post-docs, etc., in the amount of $2.5 million before we got the grant. From the institutions? From the various institutions. So, was there much administrative work that had been done? There's an infinite amount of administrative work. Do you mean cost, or do you mean effort? Well, I sort of mean both. I mean, I imagine the effort must have been great. The effort was horrendous. Finally, two years ago, I hired an assistant who is paid as a postdoc. She has a Ph.D. in physics and chaos.

1:02:30 Her name is Helen Nelson. And that has acted to tremendously relieve the administrative burden. She does things like harass people, send in their reports, and send out notices of our meetings, et cetera, which I had been doing previously. She also transcribes tapes of TI meetings, et cetera, which we then circulate for minutes, et cetera. So a lot of that work I had been doing she does instead, and it's quite a benefit. It's a tremendous effort to keep up with the reporting. The original, well, we require quarterly reports from the copianists, which are actually subcontracted from Texas. And that in itself can be a terrific headache. Also, we have a newsletter, which is monthly and requires a lot of harassing to get any contributions to it. We have a terrific amount of correspondence having to do with the funding situation, you know, will we get an extension when it's available, when is the money available, here's my budget, et cetera. So, yeah. So it was more or less necessary to have somebody working together? Yeah, it was. It was very difficult to do it without it. I did it for a couple of years, but this is one of the funding redirections. The other aspects of the infrastructure that people mentioned to me were things like the web page and e-mail and electronic methods of communication. How important is it? Well, certainly the way we do it. The web pages are very useful because you can actually see things, you can see graphs, I don't know if I'm going to find one if I try. But we e-mail all the time.

1:05:00 So, here is something discussing stability of one of our realms, where you can actually see the grass, which is interesting. the thing looks like it's settling down. It's developing a sort of high-frequency noise. Here's one that does settle down. So people would post their sort of work in progress in that? Yeah, that's right. We have a phone call at 2.30, and no doubt somebody will post stuff that we're supposed to look at to discuss during the phone call. So it's kind of also ancillary to... Yeah. Now other things like having these cameras on the video, on the monitor, et cetera, on the computer don't work as well. The Internet never had the bandwidth that we would need to do that. To sort of do video conferencing. One of the products I didn't mention is something called SCIVIZ, which is a collaborative visualization tool that lets you look at that and that is a product that was developed at Cornell, which we do use. You can actually use it over the net interactively to look at data somebody else has generated and to change the data in a way that they also see, you know, a rotative image or something and point to interesting aspects of it. We find the telephone and that works best, the phone providing the human communication. Interesting. So the head of the net and the phone are used together, too. Yeah. People on the phone all the time, actually. So with that, you'd be sort of manipulating maybe some 3D frames or something? Yeah. So I'm wondering if you think that this type of very large scale collaboration

1:07:30 is something that might provide a future model for work in graduation physics or maybe just something that just came about? That's a psychological or sociological question. if you want to get something done it seems to be the only way to do it what I mean is if you want to get something specific done it is evident that many people in the collaboration and many people outside the collaboration don't want to work that in such a focused manner on one particular problem And there certainly have been interesting things that we can and could pursue, some of which are being pursued or will be pursued that have shown up. But the amount of development that's gone into the two black hole problem in the past four years has been amazing, remarkable, and it simply wouldn't have happened if we hadn't had this. So starting from five years ago, if you had your choice between taking the Grand Challenge as it is or just increasing the budget for all relativists, at the end of five years, you would not have had much development in binary black holes. You would have had some. And you would have had a lot of development in all sorts of other things. Since you didn't spread the money around, more development in black holes and less development in the other things. Now, if you go back to a standard model, just a spread around model, I would guess that in 10 years, you wouldn't be able to tell the difference of what could happen. People who are doing computational black holes are also of the type that like to do individual research, and they're going to spread around. And so what we're going to have is a general development of the science without much evidence that there was or was not a black hole grand challenge. On the other hand, if you seriously wanted to continue to develop black hole, binary black hole computations, then if you continue to focus money into it, obviously you would continue to develop that at the expense of other areas. And this is the sociological question of how much of that do you want.

1:10:00 Now, I think that after a while that would cease to be cost-effective in any sense, the payoff will become less and less. I'm not sure that this is the time to totally abandon the effort, particularly because we can see these close ties to LIGO. But once LIGO gets underway, once we understand how to do the simulations well and connect them to LIGO, it definitely would make sense at that point You should say, well, what this ought to be is some sort of background maintenance subject for LIGO and going to other things. Of course, if LIGO turns up interesting things, and if we can simulate some of the waveforms for those interesting things, then I think the opposite is going to happen, is every effort is going to go into this kind of stuff. Now, the other thing to say is this is pretty hard. It's not as easy as ordinary astrophysics where you can just wave your hands and write equations with twiddles on them. It's very hard to make this work right. So I don't know what that means either. It probably means there will be a limited number of practitioners in any case. More than there are now, probably, but a limited number. People would have to be willing to that. Yeah, it takes a while to learn. Matt Chalkwick is the person who can teach you, though. He's very good, and he's an extremely good teacher. He has been a big asset. Well, since the connection to his projects in Lago is obviously something of his particular interest to me, and since, as you say, that could become a strong motivation for continuing to work on the field, How strong motivation was it to begin with? From my point of view, it started as a scientific question, not directly connected to LIGO. That original conversation with Richard Isaacson almost certainly was motivated by LIGO. So the NSF was interested in supporting LIGO, and the NSF was interested in supporting computational science,

1:12:30 computational activity. And those two things were connected outside. But from your point of view, in any case, that's the problem for its own reasons. Although it would have been extremely hard, again, to make the kind of progress, to make any kind of satisfying progress in the area if we hadn't had a grand challenge. because we're only now making what I consider satisfying progress and it's been four years and they're pretty much required to get all of these together that's my interest but you mentioned earlier and I gather that this was an aspect that people thought well computers are and that this is perhaps another motivation people thought computers are getting increasingly good enough that it will provide computing power to well yeah we thought so computers are not as good as we thought they would be I don't know I don't remember the curves but I think we thought we'd have teraflop computers and terabyte memories and we don't by this point Do you still think that they're liable to improve enough to push things forward or is it still mostly just a question of conceptual problems and problems of detail to work out? I think there are still going to be conceptual problems in the computations. The computers are getting to the point they will be available in the next few years. they may not be there may not be enough of a resource to do huge numbers of computations I still think that there are computational details that have to be handled but I think we're approaching that too so I think you know we're converging on that I think it's the whole subject is in fantastically better shape than it was when we started. And so you mentioned that you think it's likely enough,

1:15:00 assuming LIGO is a big success, that LIGO could provide a big business to the subject, even though it's a difficult subject that people might not necessarily... Well, right, but at least the people doing it will be appreciated and their results will be sought after. can't help a true story. Do you foresee, I know that talking to Jeff Winfer when I went to Pittsburgh, that he talked in terms of numerical relativity more or less, I'm not sure if revolutionizing is the wording of general relativity, but I mean, you know, he felt completely altering the way people looked at general relativity and providing a new way of doing relativity. If that's a goal to be sought, the tools will have to become a lot more usable than they currently are, a lot more accessible to people who are not numericists. Because numericists work slowly and take a long time and just can't tackle every problem in relativity. On the other hand, relativity is a complex system with complex phenomena in it, And therefore, essentially, computation is essentially the only way to approach it. So I think ultimately it'll be the way you find out things about relativity, not the way you understand them necessarily, but it'll at least tell you what to worry about. It expands your mind in the sense that it allows you to see things that you couldn't and then you have to think about why they act like that. So it's a great theoretical tool. I'm going to have to go. Thank you very much for spending so much time. Yeah, well, interesting subject. I'll be glad to talk to you by phone or something at some point if you wish. Sure, that'd be great. Amen.