Interview with Joan Maso

0:00 Okay, and now we are recording, and I'll just say that it's February the 12th at a quarter past five in the afternoon, and I'm talking with John Nassau. Okay. Well, I guess I should probably start by asking how you got interested in numerical relativity, how you got into the subject. I was doing theoretical physics, I was good with computers, and I wanted to do something difficult. I didn't know if it was relativity or quantum field theory or just something that sounded strange and that nobody did. So I got interested in relativity. That was in my home university in Mallorca. My thesis advisor said, hey, this numerical stuff seems to be catching up. You're very good with computers. Why don't we give it a try? And we just started like that. And it was like, there were all these American groups. And he said, I have these ideas about doing things in a different way that other people do. I said, oh, great. That's exactly what I want. And I started in 88, I mean, 10 years ago now. Just like like that, finishing physics and looking for a PhD on something. I really didn't care if it was relativity or something else, as long as it was difficult. Right. Sorry. Right. And was the idea that your advisor were talking about this happy poly? Yeah. Yeah, the idea was he had been doing his postdocs in France and he had been in contact with the French group, which is very mathematical and theoretical, I mean, Madame Choquet-Bouchard that was visiting here, was there like considered one of the big mathematical physicists in relativity. And these people, like now we have here Helmut Friedrich and Bernie Schmidt, these people have been knowing for years a lot of theoretical aspects that have never been applied to what was called numerical relativity. And he just said, well, they are not aware that all this may be useful for numerics, and they really don't care about numerics. I certainly don't care that much about numerics, but hey, now I want to change a bit the field. So why don't we look at if these like conditions that Schauke proposed

2:30 six years ago, that was in 88, and was proposing in 82-83, can be used. And we started like that, not even thinking about hyperbolic formulations, but just a slicing condition that proved to be nice mathematically. From that, over the next years, we just constructed code and then found out that we could make the system hyperbolic, which is something that now, after these years, we know that like Friedrich had already proven in 85 but we were not aware of that work but we were like focusing just for numerical purposes which is something that mathematicians I think they don't care and most of them feel like it's like they're in their hands and that's how it got started but my experience was much more numerical analysis and computer stuff than any deep mathematical knowledge and his experience my advisor was very mathematical he had done mathematical relativity his PhD and for his postdoc and he didn't have any numerical experience so it was a good a good way of interacting so that's why I have continued work with him all these years and now I'm going back to take a position there because he really wanted me to go there so he got me a position and his name is Bonner? Carlos Bonner, yes Now, when you said that you showed that these conditions, slicing conditions allowed you to construct this hyperbolic version of the field equations, was that a goal that you No, it wasn't. At the beginning it wasn't. it was it started just let's see if this is useful or not the goal was to put the equations not in hyperbolic form but in balance law put the equations in the same way as the people in computational fluid dynamics put them because what we thought is a lot of people are doing numerical relativity and they are using numerical methods that they make up they say hey we have this method and it evolves the equations we compare the computational fluid dynamics community dynamics community and we have like some friends in Valencia and that now we collaborate with but then it was hey these guys do a very advanced numerical methods why can't we use these methods because the system is too complex what happens if we can put this system in a easier way and our first work was that

5:00 just to try to put the system not caring about hyper publicity at all but just in a form flux conservative where we could use those numerical methods so it was numerically driven from the beginning. Working with that after a couple years, then we really found out, hey, if it's hyperbolic, it's much better, because then you can really exploit the hyperbolicity in your numerical methods too, and in your boundaries, and in many other ways. And that's when we tried to make it hyperbolic, but the goal was not at the beginning. So initially the goal was to try and cast it in a form similar to the hydronomics and take over their experience with their goals. Exactly. It was like, why should we reinvent the wheel about numerical methods if so many people have experience? And here we are always saying, oh, no, our equations are so complex, we cannot really use any standard method. So did you ever have any success in casting in a form similar to that of another numerical? Yeah. I mean, after, I started my PhD in 88. In 89, we published a paper where we put the equations in that form. For that, we required that special slicing. Three years later, in 1982, we realized how we could do it hyperbolic. But still, it was that special slicing, harmonic slicing, that is the one that had been used for 50 years to prove mathematical theorems and do everything with mathematical physics. Nobody had really used it in numerical work because it was not clear if it was useful or not. And we just used it, and we showed that it was useful. But then when I went to NCSA in 93, it turned out that for black holes it was not that useful. So then we decided, hey, can we make it hyperbolic, not just for this case, but in a more general way. And we spent the next couple years trying to find that. And we also succeeded. So you mentioned that the type of slicing was harmonic. So this is related to the harmonic And previously, even though that was, it had, of course, a long history in the radiation problem on the Luke's side, people hadn't employed it. People didn't make a relativity. There is even a, York in 78 has like one of these reviews in the Yellow Book, or the Yellow Book, I don't even remember the title now, Astrophysical Sources of Radiation or something. He mentions that this slicing, which is so useful in proving theorems, could be useful numerical work. But maximal slicing, this elliptic condition, was so heavily used by everybody. And a lot of work in the 80s were using slicing conditions like this that people

7:30 were not interested that much in harmonic slicing. In fact, one of my first talks, when I visited Texas, and I remember giving a seminar saying, we have this hyperbolic system for harmonic slicing. And some people were saying, harmonic slicing is no good. I mean, who wants This is useful for mathematical things, for numerics. It's not good at all. I was like, okay. It works. But so people, I think, were a bit reluctant. That's why we decided to extend it to a wide family. And then the funny thing, a couple of years ago, then a lot of other people started to produce hyperbolic formulations, and some of them only work with harmonic now. And people say, no, harmonic is perfect if you have a pie horizon model of conditions. Why should you need any more than harmonic? Well, this is funny always, so this is how it goes. So why did it turn out, when you went to NTSA, that the particular form that you'd cast the equations into to take advantage of the hydrodynamic experience, why did it turn out that that was not useful for black holes? It was useful for waves, phenomena, for many things. But for black holes, it turns out that harmonica slicing is singularity avoided in the sense that when you approach a black hole, it will stop the proper time. But maximal slicing stops well before reaching the singularity, so it has a limiting surface. Harmonica slicing doesn't and gets very, very close to the singularity. in fact so close that any small numerical error makes it crash so numerically the code was working it was converging to the right answer but you really needed to have a lot of resolution to really have something that with Maximo you could have easily more resolution than was really available you were getting so close that the metric functions Going, like with maximum, it goes to 100, 200, 1,000. With harmonic, it goes to 10 to the 30, 10 to the 40, 10 to the 50. So your gradients are so huge that no matter how much you put there, the problem is really ill-defined. There was really not much hope for harmonic and black holes. If you cut the hole, if you use this apparent horizon by the conditions where you know where it's the horizon and you evolve all with the outside,

10:00 then it's useful because you're not going to get close to any singularity and then your slicing doesn't need to try to do that very difficult work. So that now harmonic may be the slicing used for many problems again. If you actually take the black hole. If you actually excise the hole. Was that the original motivation for the maximal slicing? Or did it have other advantages that made people want to use it? It has nice geometrical properties. I mean, it's well-motivated. It's like you have this coordinate volume that you want to freeze, so precisely think the coordinates don't get out of hand. Maximalizing forces this. The volume element, in case of no shift, remains constant. So if you want to do this, it turns out you need an elliptic condition, but it has to be global. Harmonic and many other hyperbolic conditions not avoid something like this. So Maximo, in some way, I think is more physically motivated. It's nicer. I like Maximo, too, but the problem is elliptic. And at that time, talking the end of the 80s, beginning of the 90s, elliptic problems still took like 95% of the CPU time for any solution. So it was out of the question, for instance, for us, we had a small computer in Spain to try to do anything with maximum, it was just impossible. So that's why we wanted an algebraic condition, something that was, you could compete locally, your algebraic was local, and then it was fast. So again, motivations were very numerical from the beginning. I've been thinking that one of the advantages then of this local condition in the hyperbolic formulas is that it can be, makes it easier for parallelizing the problem. You don't need to worry about parallelizing your elliptic solve, which is always much trickier than parallelizing just finite differencing equations. It was really... At that time, the word was vectorized, because there were still no parallel computers. Hey, it's only 89 and 90, but there were no parallel machines yet. So vectorizing was very easy then. And one of the things, I visited 10CSA in 92, and I was very happy to just get my code, that it was like a C code, convert it to 410, vectorize it, and get wonderful performance, which is something that, hey, Ed said, hey, this is impressive.

12:30 It was easy. And the advantage was this algebraic condition. Then the CN5 came to NCSA in 1993. Again, a code that I developed extending the one for my thesis was one of the fastest codes in the machine because a combination of really thinking a bit on how the architecture of the machine works and then having this condition makes it really wonderful. and the codes using the maximum condition were much slower. So even prior to the advent of the parallel machines, the ability to vectorize it meant that with a single processor you were allowed. Well, is that the single greatest advantage of the hyperbolic form? No, I think that was why we started. I think that that is why still we really pursue some of the aspects. hyperbolic formulations is not just now anymore because of numerical issues, is now much more because it has brought the connection of the mathematical people with the numerical people in the end. Some of these other hyperbolic formulations are developed by theoretical people working with numerical people, like York, with Choquebruja, and Abrahams. So the two communities now understand each other's problems, which is something that before I think they just ignored what the other community was doing. Right. So now people in the numerical community are aware that having a mathematical problem well posed with nice theorems about existence and unicity of solutions is something good, is something that you have to care about. And maybe 10 years ago people really didn't even stop thinking about this. so their experience with the usefulness of the hyperbolic codes has sort of taught the numerical people to appreciate better the input from the mathematics and at the same time you were saying that the mathematical I think some of the mathematics now are standing to look at numerics with interest and saying hey the Einstein equations can be solved numerically I think that a lot of people still had the attitude 10 years ago many of them do now of these numerics

15:00 I don't trust it. I don't like it. It's not worth looking at it. But now we have... Werner Schmidt comes to our group meetings, and he's here. He's interested in what we do, and he asks questions, and he always puts problems. Oh, you don't do this properly. At the boundary, what are you using? Well, we have a condition that is not really what we should be using, but we get away with it. Oh, this is not good. and Peter Humner who is the guy I was talking with I don't know if you have talked with him or not Peter Humner has a very mathematical background and he's now doing numerics but not in the numerical relativity group, more in the mathematical group with these also hyperbolic equations but in some way with even nicer properties from Helmut Friedrich and he's the only one now trying these conditions but it's so much more difficult from the mathematical point of view that formulation that a lot of people in numerical relativity, I think, are waiting to see what happens there. I don't know if you have talked to anybody about this or not, you know? No, I understand. Helmut Friedrich has been working in the last 20 years in mathematical formulations of equations that are hyperbolic trying to see and also that do not cover like a finite amount of space that you don't have to put another boundary but that the equations are recasting you do a conformal transformation so infinity is in your grid and you go beyond infinity that infinity is a scry So it has a very good way of defining where is scry, where is no infinity. Reading information, physical information, at infinity when the space time is asymptotically flat. You have everything there. You can read all you need. And that has been, for the last ten years, something completely theoretical. Now at least one person, Peter Hüner, is starting to try this numerically. in 1D, now in the 2D code, I still need a couple of years of testing, but I think it looks very, very promising. I'm definitely very interested. So these are sort of compactified grids in which... Just not compactified the grid, but in some way, the equations are not just the Einstein

17:30 equations, but are the conformal equations that are a superset of the Einstein equations, where you have more freedom to choose how you map this conformal space to the physical space. So you have physical space time and the conformal one. But I don't want to be the one explaining this. Sure, sure. If he was hearing me now, he would say, hey, this is absolutely crazy. This is not properly explained and blah, blah, blah. It's funny. The interaction here with some of the magicians, not the magicians, is sometimes pretty funny. So obviously the two groups still have different sensibilities and different kinds of things. So now the CACTUS code, which I guess you've been very much involved in, it implements both the hyperbolic and the ADM. Hello? Okay, so there we go. So let's see, I forget where we were. Let's go back and check. We could listen. But Cactus, you're asking me about that. Oh yes, that's right. It has the hyperbolic formulation and also the standard one, so we can do comparisons and give the people the opportunity to use both or do whatever the hell that they want with it. So is the situation more or less at the moment that it's difficult to tell exactly what method is going to be the most effective? I mean, the hyperbolic formulation that the code has now in 3D doesn't take advantage of these numerical methods from the composition of dynamics that really take advantage of the hyperbolic part. It uses methods that take advantage of the flux conservative part, but not the most advanced ones. That is work under development. You can have a flux conservative form where you can use standard methods like McCormack, Glatz-Bendroff.

20:00 These methods get you something, but if you really want to do the best you can, is when your system is hyperbolic, you have this matrix of fluxes that you can diagonalize. And instead of evolving then the standard variables that you would evolve, the metric and the stringency curvature, you evolve combinations of eigenfields that propagate in a certain direction. This is the big advantage of hyperbolic formulations. Then you can apply the methods to the variables that go in one way or another or do things special at the boundary. And this is the big promise of this in 3D. It has already been shown. We have shown in 1D that it works beautifully. And comparing hyperbolic formulation with these methods and with this treatment to ADM, it's vastly superior. But we have only managed to do it in one dimension. Now we have to do it in 3D. And we have been working already like more than a year on it. And we still need many months to try to see if we can do the same problem now in three dimensions. But that's the goal. Meanwhile, what we do is just compare it with the ADM formulation with very similar methods, because the equations should be very similar. At least the Einstein equations transformed with constraints. The physics have to be the same. You are not really using any advantage of a boundary condition or a numerical method that is vastly superior, so we're getting very similar results, what we expected but we have to prove it I mean it has to be shown that the formulation is as valid as the standard one so this has been the work of the last months just to show that in 3d one is as good as the other now the work of the next year will be to prove that the other our formulation is better because of the numerical methods and the mathematical stuff that you can put in the boundaries. But that's a promise. I mean, it may not work. So with the hyperbolic formulation, do you need to excise the black hole? In one dimension, we have not needed to, because we have used methods from computational dynamics to treat very big gradients. We have used maximal slicing in that case, because now our hyperbolic formulation can use maximal, something that we couldn't do before.

22:30 There we go. It's a lot of tough stuff, so it can get confusing. We were saying about exercising, you were using maximal slicing now. We were using maximal, so the gradients are still very big. With ADM formulation, one can evolve for, let's say, 100 or 200M in one dimension. A black hole with certain initial conditions and a certain boundary are close to the hole, trying to mimic what you will have in 3D, a certain resolution, not to try to put 5,000 points in one dimension, which then you can solve the problem very well, but putting 200, which is something, or 400, something that you will have in 3D. The standard formulation, the ADM, runs for 100 to 150M, and then it crashes, because the gradient in the metric function, GRR, is too big. With our formulation using the same as slicing the same boundary, but just add the boundary Putting the right Stuff that has to come in and using the right numerical method we evolve for thousands of n and The peak in GRR goes not to a hundred but thousands and it's not a problem But we would love to do this in 3E our first tests are not yet thousands. They are now the same as the ADM one, so that means we still need a lot of work especially at the boundaries now. Now it's more a question not of numerical methods, but more of mathematical insight of what to do because in three dimensions, if you reserve your system to spherical symmetry spherical symmetry doesn't have gravitational radiation so the boundaries that you can design can take that easily into account So I have pure gauge. I can't reduce this gauge at the boundary. In three dimensions, you have all the degrees of freedom. So you have to pick up what degrees of freedom you don't inject when you have a black hole. You don't want to have incoming radiation. And this is not a trivial mathematical problem. So we're working now in collaboration with more analytical types. And more in my home university, there is a guy who has been doing symmetries in general relativity, studying just the problem of symmetries and transformations for non-homogeneous cosmology.

25:00 And he's tired of it. He's doing this for 10 years. Now he sees numerical relativity as a good ground for work. And he is now starting to collaborate with us in this problem. he's got interested in the outer boundary here is not enough to just say I put it very far and I use brute force which is what we use now I mean now we use brute force. Put 300 points if you can and put it very far. We need something more clever. The other approach that I was telling you Friedrich avoids completely this because the outer boundary is really at infinity so you don't have to worry about boundary conditions The other thing that a lot of people are interested in is adaptive mesh refinement so you have boxes and boxes and boxes so you can have like fine grid here, a coarse grid and a very coarse grid so this goes to very very far and then you don't have to worry about boundary conditions either because everything can be left essentially static and this is work that Lee Wilde has done here and now other people, Tom Goodale and others are trying to join in the Cactus Code. I mean, the educactoscope, in the end, will have so many things that we have to try everything that will possibly work and see what actually works. Well, I guess one of the big elements in the macro relativity, well, in the binary black hole problem nowadays is the possibility of producing waveforms for gravitational wave detectors like LIGO and so on. At what stage did that sort of become any kind of a factor in your work? It has put pressure over the last couple of years. It puts pressure. I think sometimes it's not very good pressure. People get to be nervous. People want to jump too fast. This is my negative assessment about what the pressure is putting. I understand why we want it, and I understand that pressure. We really want to get out of physics. We really want to do something that is not a mathematical incorporation of non-linear space. We are physicists in the end, but I think some people are getting nervous and trying

27:30 to do some problems before they solve others. I would say that over the last year we're finally striking a good balance between how to proceed and not to try to do the two black hole problems before we understand better the one black hole problem and real waves and other things. But the grand challenge was a ground for a bit of tension because sure I told you there is tension between groups more like political tension about who gets credit and things like this and who wants to do it his or her own way and I want to do it this way no no you're wrong well I don't care if you say I'm wrong theoretically we were collaborating and in the end there was not much a collaboration so that was a bad influence I think and those Those couple of years didn't stall the process, but could have been much better, I think, without the Gun Challenge. Now I think the Gun Challenge is finally getting their act together, because they are smaller and only two groups really working together, which is much easier than eight, and now they publishing stuff that is worth doing. So, from your point of view, if I understand you correctly, one of the problems with the grand challenge was that the goals were set too high, too high, unrealistically high, saying in five years we'll solve this problem without ever having attempted many of the steps that you need for this problem. It was formulated in 1992, a year when the state-of-the-art two-dimensional code, the best they could do for a collision of two black holes was to evolve up to 100 or 200 M. Now we were talking about the coalescence, merging of two black holes that you need to run for 10,000 M in three dimensions? Come on. The apparent horizon boundary conditions were just formulated that year, essentially. Hey, I just thought that was absolutely crazy. Maybe thinking that you can put very smart and bright people, and you have many of them, they will do it.

30:00 Maybe it works, but maybe it doesn't. I don't know. You're asking a lot. And I think that they asked a lot. Was the... I'm talking to Paul. He spoke of the collaborational infrastructure idea in the Cactus Code. So was the Cactus Code influenced by your experience with the grandchild? It was. It was. Characterscope grew out of good and bad experiences. Good experiences about opening a code to collaborators that I had developed at NCSA, and people there and people at Washington University were using, and I was using it for people in Spain. But it was like a code specific for the CM5, so it could not run anywhere else. then we opened that code to the computational science community to help us to do nice demos demos of virtual reality and things like this but that was a special version that didn't have things that the other version had that experience showed me that hey having very different versions of the code one that used for this, the other then they don't get merged together, people work that was crazy The other was, looking at the grand challenge, all the fight about different codes and copyright, and I want this, and this is mine, and this is yours, and you cannot get it onto this or that, was also showing, hey, the model here is not really working. And the truth is, it has taken us a couple of years to realize what's the best model. And I would say that now, we finally think that the best model is the more open one. The more open, the better. Give everything. Don't even think that, no, no, no, I will give you almost the best I have, but I will keep the best secrets for myself. No, no, no. Publish your work, give everything. Anybody can take it, reproduce it, and see you're right or you're wrong, or do whatever they want with it. I want to release Cactus under the GNU public license, so it becomes like the Linux of the community. I mean, a year ago we thought, no, no, we have these groups and we will be code for our groups, not for people outside.

32:30 But then if somebody from outside comes, then we will give him special permission or her. All that started to get really tricky. I mean, who is part of the group and who is not? What happens when a student is here, finishes a PhD, and goes somewhere? Does he or she take the code? or other people come, I mean, we were starting to get, like, paranoid, a bit paranoid about is this open or not, and it was like, hey, no, after many conversations with Paul, with Ed, with Schutz, we decided it has to be more open. Over the last couple of months, we have really come to, it's even more open than we could even imagine at the beginning. So part of the culture of the numerical relativity community had been that sort of, you know, people would hang on to the best bits of code, for example. I would say this is the standard practice now in the astrophysical community. People have their own codes doing hydrodynamics. They are very advanced. They are very, very personal. I mean, when a code is finished, that code is used as a production machine for many years where that group produces papers with that code without ever sharing any of the infrastructure of the code. It's funny. In that community, four groups have four different codes that produce four different results. Everybody fights about saying my code is right, your code is wrong, my code is wrong, your If the codes were open and everybody had access, everybody would be able to say, hey, your code is different from mine because of this. Because when you publish things, the details are not there. I mean, if people really, when they publish something, it would be okay if they really say 100% what they have done. That would be equivalent to publish the code. The other person would only have to code that up. But they don't really do this. They keep the secrets. But in part, it's understandable. productive life depends on this. They know that once they get the code set up they will be able to publish a lot with that code and publish or perish. I mean a lot of people think this and students, postdocs and even professors are always obsessed with publications

35:00 and they want to publish, they want to publish. If you have a code, publish alone. I understand it but I don't like it so how do you envisage cactus being used by the wider community have lots of people do you think would be using it if you you know make it available as you say I hope that many people in the numerical community will use it I am I envision not powerful people using like small groups in third-world countries interested in numerical relativity. And now we have collaborators, a collaborator in India, interested in CAPTUS. Now a guy from Venezuela is visiting in two months and he's talking with me about using CAPTUS. He has a group of eight, six people trying to develop their own code with message passing, with a machine with eight processors. They spend like two years of development and I tell him, hey, we have this, why don't you Do whatever you want with it. I mean, you can modify it as long as you say, hey, the original version was by these authors. This modified version has this. Well, I can do public license in this sense. Do whatever you want. People then are like, hey, this is great. I hope that it will start like this. People who didn't have the resources to build big codes on big computers now will be able to run on big computers or clusters of workstations. and then eventually people in the states with big groups will be also interested but maybe at the beginning it will not be easy to convince them to just drop using their codes i think that if they realize the code is open a lot of people are using it the results are published there is a web page with the code with the results with the data completely transparent where anybody that sees a paper can go and reproduce that data and change something and see if they can improve it and then publish a paper on that, maybe the community will realize, hey, this is great. It's a dream now. We'll see how it works. I'm excited about it. I really think that this is one of the best things that we have done, if it's been not also intentionally at the beginning.

37:30 That sounds very interesting. Talking to Paul, you were saying about that there's a committee to oversee the integrity of the core code. You were saying that there would be a slightly different model with the wider distribution that you envisaged with the GNU where people would be taking the whole package and then modifying it. Would there still continue to be an inner group within the current collaborations that you have going which would be all using the same I think the committee now is like an interim solution for consistency and for checking that we're doing the right things with now the work of many people because the code has been used at the beginning only by three, four people in the first couple of months then more people, then more people and now we are like a lot of people using it and now we're discovering things that we never thought that people need Even if Paul is an amazing guy, he could sometimes not code something and not say, hey, this. And I was more like helping as a designer more than coding. I did code, but he really deserves the part of the credit for doing most of the hard work. Because all the chunks of code that I put was the stuff that I had already coded before for all my codes of the last year. So for me, it was more a continuity process. a lot of work in an intense period of time. Now we see that people ask, could you support this? Could you add this? Could you add that for the people here? The committee now needs to be consistent about what is worth adding for the future and what is a mistake, because you may add something that then doesn't have any way of being useful for other people, and it requires a lot of work by people here. so we have to prioritize what we do here we don't have infinite resources but I think that when it's open there are a lot of people then hey if anybody wants to do something I mean doesn't need to ask permission from a committee I mean he or she does it and then says it works do you want it and we say yes or no I mean depends if that's useful for you so I think that model will be in the end more useful than any committee or central person deciding things Yes. But for that, it needs to be widespread use. I'm amazed that now it's growing. Fifty

40:00 people or something. For a community so small, that's a lot of people. that's right it's not a very big community yeah exactly I should probably let you go whenever you want we can keep talking without recording yeah we can talk as we go