Baking a mathematical pudding: the role of proof & experimentation? (contd.)

0:00 My name is Ivo Hanze. I'm from mathematics education and I'm working at the University of New York in Germany. This is all I'd like to thank the organizers for inviting me. And when I got this invitation about 18 months ago, I think it was, I sent this title to the organizers, and it's a very long title, and if you look in this room, it's a long title. And 70 and a half months later, well, I found out I promised too much. So last Sunday, I really became aware it's clear it's too much. So first of all, I delete some words. I don't want to speak about possible implications for the mathematics classroom, because right now, I mean, well, it's difficult for me to find some implications, really implications for mathematics. But I added the subtitle, the presentation is in the description, and I hope to get some answers from you after the talk, so I'll change the rules of the game. You have to answer my question. Just a second. A second remark before I start my talk. I am a math educator and as a math educator my goal is to investigate teaching and learning programs for students and children. On the one hand. On the other hand, I want to create learning environments for students to learn mathematics. And create such learning environments means that I have to take a lot of knowledge from other disciplines. Together from psychology and so on. And we've predicted that from the philosophy of mathematics because if I want that children to learn mathematics, students learn mathematics, I have to know what mathematics is. This is very important in such cases like uni. There are a lot of teachers who have very restricted image or belief about groups. Groups just show that something is true or it's correct. To create a helpful or successful learning environment for this group, I have to investigate what this group can really practice in mathematics. This approach is not new. Since 2003, we have a nice name for it, Mathematicology.

2:30 Dirigat Delfler wrote a very nice paper in the educational studies, and he wrote a mathematics education on how to study and investigate mathematics. As it is currently practiced, it is produced and used in all its forms. This is not a mathematical study, but it is a meta-study. And this is my motivation to look on now the social uses of accepting new mathematical results. Okay, I shall tell you about... The conscience, however, gives some ideas about mathematics as a social product. A few minutes ago I learned that this is not common sense. Well, then we'll talk about the process of accepting the results. I have another question. I want to get some answers on this question, so I ask mathematicians. I made an empirical study with mathematicians, so I get a question mark, and I will present the answers here, and we'll discuss the answers. Well, and then it's a question if there are implications from these results in the mathematics classroom. Till now, I don't know. So mathematics as a social product. Here I only will give some issues. On the one hand, we know that most part of mathematical theory is not formalized in a relatively detailed way, based on axioms. That is the fact formalized means here, formalized in such a way that a computer who knows axioms in some groups of applications can check it. But even if we have radically harmonized truth, then we don't know if our theories are true, because we don't know if the axiom system is inconsistent or not, or not. So the way to check if groups are correct or not correct is not a question of how to do it in this way, it's a question of mathematicians finding agreement when to accept a group and when not, so it's a social question. Such a group is just that the group's reliability does not primarily come from mathematicians only checking documents.

5:00 It comes from mathematicians seeking carefully and critically to develop mathematical techniques. And I think you all know the famous sentence, the proof is proof after the social act of accepting it. Well, I think you know this man, Lappertort, who is quasi-theoretic and theoristic view on mathematics. He added another component, an experimental component in some sense. He said, okay, the result is true within a context, it's socially accepted, but if this context changes over history, then maybe an accepted proof is not accepted anymore. He gave some results, this may happen, and he gave some examples, this may happen for other theories. It may not. For my part, I don't consider this historical research. I only deal with the present context. So, the proof is a proof after the social act of accepting it as a proof. So how do we accept a proof? That's the question now. If we consider natural science, theoretical natural science, experimental physics or chemistry over data. Then we have the naive idea that there must be an independent replication of an experiment and a result. So the same physicians, another researcher, collect the same experiment, get the same results. And so the result is replicated and the result is accepted. In social science it's a little bit more difficult. Here, we also need an independent reputation, but more than one. Here we have the case for example in education research, is quantum-based learning more successful than direct instruction? There are a lot of studies. And maybe after 10 years, one person says, okay, we have 100 studies, now I'm going to do a meantime analysis, and then we can look, 90% of the studies said, okay, problem-based learning is better, 10% have other results, so we may see problem-based learning is better.

7:30 Last year, in the American Psychologist, I think it was... Jennifer Pike published an article about the meta-analysis without meta-analysis. She investigated gender differences and she took about 100 meta-analyses and made meta-analyses. And the result is, in most cases, men and women are more similar than each other. So we have something like meta-analysis or meta-analysis. From my own experience, I made my PhD in algebraic computer science. I use computers to compute proofs, to compute economic attitudes, graphs, to compute subgroups of these groups and so on. I know that in this case, we also need the replication of computer-based results. When I presented my results in conferencing, people asked me, well, nice results, but have you checked your results by another computer system? So I used the computer system GEP from the University of Aachen, which is now in St. Andrews. And then I took another program, which was programmed at the University of Moscow, and everything was calculated and computed again, and I was lucky I got this. The people said it must be another, that was just in base and other I've written. Okay, what about mathematics without computers? So, let's try it again. We define the reading and the understanding of a computer as an experiment, but in the sense that we design or whatever, just fine. And the evolving individual conviction defines the result. And we can say accepting a mathematical proof means you determine the location of the theory and the result. Question? How many notifications do we need? We have something like, similar like meta-analysis in mathematics, the big model, the string slices. Unfortunately, we do not have a general annual confidence in mathematics.

10:00 We decide every year which theory should be accepted in mathematics. It would be nice to have such a conference that we have no problems with. It's not a joke here. In other disciplines, we have such conferences. For example, the general conference on weights and methods, they define what a meter is, and the definition of a meter changes over time. But in mathematics, we do not have such a thing. So, the question is, who determines which theories and which groups should be accepted? The mathematics community. And they decide in a social process, negotiation or whatever. Who is this mathematics community? All mathematicians. Okay, but how can you describe the process of accepting the theory? How does it work? Do we have ideas about it? Or the other way round, what has a mathematician to do that his or her theory includes acceptance? My wife is a mathematician. She has a nice reserve. What's she supposed to do? The very results aren't accepted. Every year, hundreds of thousands of theories and truths are published. Yes, I think. Hundreds of thousands. Thank you. How many are really accepted? Davis wrote in 1972, most groups who research papers are never checked. I don't believe so. I think most groups at least once were checked. All groups of PhD students are checked as a quorum, when I prepare my final look on this group. Okay, the mathematics community is a good community, and I don't think that this community as a whole will accept or reject groups. Mathematics is a degree which is divided in a large number of highly specialized research areas and generally most mathematicians have specific knowledge in their research area and something like a basic knowledge in other areas of mathematics.

12:30 Now, most theories and groups are only interesting or sometimes even understandable to some mathematicians, I think. So, the accepting process for theorem and proof is mainly associated with a small period. I had a Ph.D. in discrete mathematics and a degree of mathematics in mathematics, working in discrete mathematics, discrete tomography. Every evening, he would ask me what he had proven, what he tried to prove. Mostly, I don't know the answer. So it's not completely different, but it's another research area there. Other objects and so on. So I'm not able to follow. I can make some general remarks, but I'm never able to check very well. So I think we have the social process in a small peer group. If the peer group of discrete tomography has decided to accept the result, then the peer group of stochastic or unstressed theory or whatever has to trust this group of discrete tomography that the result is true or not true. So the acceptance process within a smaller peer group is a little bit more easier, I think, because there is more communication, more personal context. The criteria and norms of physics mirror. So if you are working on these three technologies, my right to do it is to come up with some classic crystals, there may be 1,000 institutions all over the world working on this field, and maybe 200 of them just working on the same special area like she's doing. So if you go to a conference and then you've got maybe 20 or 30 of these special conferences to speak about there. However, the only question, what does process of acceptance mean in this community? How many replications are needed? Is there a moment in which acceptance occurs? Today, the theory was accepted. Or is accepting communication a process?

15:00 So I'm saying, more and more mathematicians accept the dignity system. I think there are some necessary conditions for acceptance. First of all, the theorem which is proved must be published. Otherwise, nobody knows about the theorem. And the second is the theorem which must be reviewed in some sense by other mathematicians. So we have to check if it's true. For the publication, we have journals. We can correct briefly and put it in the web to the Archive or wherever, conference talks and so on. And the revenue process. Well, we have peer reviewing journals, but the criteria of reviewers are sometimes a little bit strange. We have reviews by peer group colleagues. Maybe we send peer-reviewed results to another colleague, but you know it's the government, not the world, so they send them as a reward. Or we have something like an implicit acceptance, which means I find my theory cited in another paper, and I see, okay, one colleague has used my theory, so I think she has accepted my theory. In 1983, most mathematicians accepted the theory that some combination of the following factors is present. First, understanding. So, people have to understand mathematics. A mathematician has to understand the theory, yes. Second, the theory has some significance. Third, the theory is consistent. With him, we are in theory. Fourth, the author is an expert. Third, he is not a human. And fifth, there is a community in which we can practice theory. I am personally not aware of the typical data regarding this question of acceptance of theories by mathematicians. I know there are some case studies, some interviews, and I know that some mathematicians wrote down their thoughts,

17:30 but I don't know if there is or exists an empirical investigation that someone has asked 100 mathematicians what they think of those theories. To do such a study, first of all, it's a more pilot study, and I asked my study which conditions are sufficient for mathematicians to accept this very historic second research question, and the differences between theories from the specific research area of the mathematician and theories from other mathematical areas. And the third research question, which conditions are sufficient for mathematicians to accept the solution to help reviewing research papers or journals? The third question is important because if you are doing a review for a journal, then you act as a representative of the unit. So the question is, there are different criteria for the mathematicians. For this small study, I used an online questionnaire. The short online questionnaire. So the reason is the following. Mathematicians are very difficult. And if you ask mathematicians whether they have time for a one-hour interview or whatever, they would say no. If you send a questionnaire to these 10 pages, I think maybe people certainly would answer this questionnaire. So I made a very short questionnaire and an online questionnaire. They can answer it by clicking. They get a lot of... Statements are shown to you and they are just like here. Here are the statements and they have just to click here to answer these statements, to rate the statements. So it was a four point scale. We have to rate the given statements as we always, frequently, sometimes, and ever. I will show you the statements in a minute. The sample I got. Answers from 49 scientists from the universities of Augsburg and Munich, 15 professors from so-called private centers, these are people who are leading professors, and 25 international students of course.

20:00 I sent this online web questionnaire only to the University of Augsburg where I worked for three years. I left Augsburg one year before. And in Munich we are working now, so there are personal contacts to these people, and I'm sure that they are not joking with me. My wife sent me this questionnaire that's linked to this online questionnaire, and we are working with the technical team in Munich. So there are two personal contacts, and we hope that from now on this will work out. Altogether, I think about 200, I guess 200 mathematicians got this link, 47 answers. There's one mathematician, number 41, who wrote an email and said to you, please, to answer this question. And he wanted to explain to me his view, and he wrote a long email. And just yesterday, he sent me a song, number 42 answers, but it was too late. So, which items these mathematicians had to raise? First, the situation was the following. I asked for sufficient conditions to accept this union in everyday mathematical world. This was important because every day was a mathematical world. It was not a question for grammar, theory, or computer or human, whatever. It was every day a mathematical world. I accepted a theory quiz. The theory was hard to be fulfilled when returned, so they had to wait, although sometimes we could be hired. I checked the key arguments of the quiz. Other mathematicians used the theory, taught the theory. I know that there exists a published proof for a long time, and there is none on the literature. The theory is used by colleagues with high standards. The proof idea was... The theorem is cross-selected. The theorem of the group of factors in the theorem is one, but I did not check the proof. The theorem is consistent with the existing theorem. The theorem comes from a pen and stone. I checked the proof immediately. The same statements were brought to Almeida, Ferrier, and Kassel.

22:30 So, regarding the review process, these statements. I checked the proofs step-by-step and understood that the theory comes from a kind of public review that gave us the proofs possibly, and I checked the key affirmations also. Now the results, and I prepared some copies for you. This is the result of the slides, and it's easier to discuss about it, although it's a bit small. So I think that's the first thing I'm going to talk to you about in just one minute. So can you please consider the first diagram? Oh, there are four different... So this is the area where these mathematicians live. And we'll be there for the first part, so they know everything in this research. We have five parts, so it's small and complicated, so it's only like 20 copies. Maybe you can take one copy and put it in. What's a Monday night or a Monday night? Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. Monday. So we have five statements which are very out of time used, at least line three, and we can see the highest rate in which I checked the proof in each of them, so I expect that this is the right. The second, I checked the key arguments of the proof, and then published proofs of this for a long time, and there is no contradiction here.

25:00 Some of these types of theories are used by colleagues with high standards. And then on the fifth grade is published in a period in German, 1906. And the others are a little bit lower. They are only on the level of class. Now I divided my small sample of the type of study in the senior and junior master's degree. Senior means PhD students or... The upper tendency is that the junior mathematicians are much more liberal in their accepting of the theory, whereas in the four cases we have statistical significant differences, the other cases are not. In the statistical sense, I think it's really a small percentage. If we consider our five guys from the slide before, again we have, I checked the tools in detail, with the highest rating, but then the second for the junior mathematicians is, there exists a publishing group for a long time in their countries. For the senior mathematicians, I checked the key acronyms. To the institute we have here, published in a peer-reviewed journal, I did not share this with you, for the junior mathematicians it's more than 50 in the new world, for senior mathematicians it's between sometimes and frequently, so the senior mathematicians are much more skeptical against it in this sense. So accepting theories in their own research area is the main point for mathematicians in this sense. Is it that they check the proof by themselves or they are sure that other mathematicians with high standards or many other colleagues, which means there is a proof for a long time in your foundation, checked the proof? So doing checking the proof is the main point. The theory of the process of learning has a certain reputation, which is known, but at the level of the human mathematicians, it is more skeptical.

27:30 Unimagnetism is a molecular process of acceptance, and the other factors, like commissivity, distributivity, group compendence, quality of thought, and the true value of justification of this new diagram of institutions, play only a minor part in our dissertation sometimes. Now, if we distinguish the non-desert area and other areas of research, we get this diagram. In general, the other research area is a little bit higher, which means they are more liberal in the other research area than in their own research area. There are three sections. I checked the proof in detail and I checked key arguments. They trust their own checking more in their own research area than in other research areas. I think it's plausible, because in other areas I don't know very much, okay, I've checked the proof, but I don't know the truth. We have only one significant difference here. It's the ear community in China. So in other research areas, they use mathematical results in other research areas. They more likely to accept proofs which they're not interested in. Maybe here we have something like an instrumental use, I think, as a series. Wouldn't you expect 1 in 10 to be significant by accident if you're doing it at the same 10% level? Well, I mean, you say that 1 is significant. You've tested 10 things. What level is it at? It's at the 10% level. No, it's at the 5% level. But it's still pretty... we'd expect... It's a 5% level and so we have only bought the sample of n equals 40. It's a very small sample and to get near a significant difference is very... So I think the sample is bigger than these differences would be significant.

30:00 Okay, acceptance hearings in other research areas. The criteria for acceptance of a series of other research areas are very similar to those in the undergraduate area. Significance difference in the question of the EF learning process. And again, the other factors like consistency and theory and so on. Second question about the learning process. Do the EF institutions act as where they are as a representative of the mathematical theory? In this case, the sample is not 40 because some PhD students didn't put in these items because they wrote a comment and said, well, I never made a review. But I know that there are a lot of PhDs with all very good reviews because they get the reviews from their professors. This was in my case too, and my wife wrote a lot of written, but always on that topic. So here is the field picture. I checked and understood the field step by step. It's nearly, it's always level. And I checked the key arguments, it's nearly incomplete. And the other statements there, we want to sometimes level off. The theory comes from the Ames College, it's nearly never. So this is not the criteria. So, again, the main point is, let's say, the mathematician checks the proofs as such, other factors play a minor role, and here I check, again, the differences between senior and junior mathematicians that I initially know, so as far as they are the same, they are the same. So how can we agree to these three numbers? Again, The sentence of Thurston, mathematical truth and reliability come about through the very human process of people thinking very different ideas, criticizing one another, and independently setting things up. So, it seems that this way of checking the truth is important for this.

32:30 My point of view is, results indicate a tendency that the clear main criteria is that people are tricked by themselves. Or, my colleagues say no, with high standards. Or, that the proof was published a long time, which means it was probably checked by a lot of other mathematicians. So you can see the results, really, it's a checking. Irregular, they said no, but particularly senior mathematicians did. Now, what are the consequences for the decision of the social crisis? My initial question. Such a process of accepting new theories includes requesting them. So it seems the ideal situation is that everyone takes a piece by himself, by him or herself. Mathematician number 41, who refused to fill in the lecture, said I'm broken. In principle, I must be able to prove each theory I use. That's what I tell my students. An authoritarian group serves the theology but not the math. They are responsible for everything they write. Each mathematician must rebuild his mathematics and use it for himself. This sounds like complete dematerialization. Every mathematician has his own valid mathematics. What does this mean for the human brain? The key group will accept these mathematical theorems, which is an intersection of what mathematicians learned in this period.

35:00 As I read him, he's not actually commenting on acceptance, he's commenting on intellectual responsibility. So I don't think you should draw the conclusion about acceptance from what you said. It's that one is responsible for everything one uses. And that's independent of what I accept. I accept many things that I don't use. I think this is only a part of this email, so only part 9. This email was about pages, and we explained everything in detail to you, and the main thing is really this last sentence, which mathematicians and mathematicians use for themselves, because he argued that to understand mathematics means to understand the proofs for the theory. Like Michael, who told before, theories are only the first thoughts. The knowledge is the truth. So if I want to understand mathematics, I have to understand the truth. And I have to work through all of it. There's only one question. I have several points actually to make, and I first of all think that the project that you are engaging with is a very interesting and substantial one, because actually there is a phenomenon in mathematics which deserves explanation also of the social law, even if there are other explanations. Why is it that the consensus about the validity of a group is so high? As opposed to other subjects. And that's of course a phenomenon, for instance, that Hina Eid has written about, and she points, or she thinks that one of the areas that we really should look at is the training of mathematics. Because they are trained into a kind of behavior that produces, in some cases, these very high-level consensus.

37:30 That's a small comment, I think, but it was just my opinion. That's my opinion, too, after we have analyzed it. And that's why problems to find implications for the mathematics of physics, that's what it means. We have to train all our students. The question is, what is it in the training of mathematicians that leads to this effect? But then, maybe a few proposals, how to push the study a little bit further, because one of the points that I could raise against the inquiry that you have raised here is that it shows you something about the self-perception of mathematics. I would like to point to some fairly obvious stages in which this sort of process must play out. A sequence of filtering steps before something is accepted as a proof and I would distinguish, I think that's a fairly simple observation, and there's an oral filter in conferences and workrooms where mathematicians work together, in institutes, and there's a very elaborate oral shock in which these proposals of the rules are. Check before they even go into a publication stage. So what's the role of this oral speech? And then the next issue is what was the role of the publication stage? And that's where peer review and so forth comes in. And then of course there's the process of what happens after the publication. Obviously we can point to many examples where the decisive filtering process only happens after publication. So I think we have to distinguish these three stages and we should try to find out what does that mean in each stage, and I would make a case for the fact that the oral stage, the first stage, is actually one of the most important. One of the things that probably needs to be understood is what the world appears to be in this oral culture. The oral culture is actually the journal.

40:00 So what's happening in UD, what's happening in the MSRI, what's happening in the World Fund. These are places, I think, extremely, extremely important for sustainable policy. I mean, you have the peers, the network of peers, which also, maybe in my last 20 years, somehow bridge the gaps between these highly specialized branches of mathematics. There are always a few peers around who bridge the gap to the next specialty. So that is almost a trite age in its traditional analysis. Not only mathematics, but also physics, which is not as nice as Newton's writing, but my mind is actually totally different. In fact, this socialization process is a very important thing. So, just to give you an overview of the process, the whole socialization process, which includes the use of language and discourse, and many other great and interesting pre-assignments, which are one of the facts that we are missing at the time. This shared, increasing knowledge which is the basis of many judgments about theories and truths. I think I could make a very extreme comparison with some of these priests who had came to a certain religious school, certain beliefs.

42:30 So we have a certain set of theories and also arguments about the religion, which for someone who comes from Alzheimer's can be an un-understanding consequence. But I'm going to explain. But this social construction of reality, I think, is what all of us have to do with the religion, even though the religion is not much of a business, I should say. But I should say... This can't be the whole story. I think because the same processes are available in different sciences, so yes, there is also socialization of physicists and so on, but so the difference in mathematics is much more... For the second modality, for example agreement, it must be related also to the topology itself, and that's something that we can hide, that she does not look into the mathematical topics in the processes, and I want to put this in the scene where comes evidence from, that there is a shared, passivable basis of the mathematical theory. I think one should not forget about, so to say, something which is not event in the metaphysical sense, but in a very concrete sense, the object of mathematical activity. And I think we need it to explain the physics of these processes and the social processes, but not just the social processes. They will not do their own work. And coming to the third one, I think you can easily explore them. In this socialization process, the students in the first semester at the university, and I was in the first semester, we had to prove that, or in a proof that appeared, the equation minus 1 to the n equals 1 to n. And we discussed with some other students and they said that proof is by induction. And I said, we learn this in school about how to deal with it. So I think this is such a question. I have several comments. You described the way in which you managed to gather sets of mathematicians to develop a question, and somehow it has to be working with all of them in the same section.

45:00 Or they are locally in the same environment. They are in the same institute. I asked them in which research field they are working. And sometimes they are in the market. Did you include him in your 48 departments as well? I was. No, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no. Subjects have the same social process because this is an assumption that is given to them and somehow, you know, we shape the culture of mathematics by assuming it and by putting everything in the same box and analyzing it. But perhaps it would be interesting to think whether it would be the case that in different subfields... The process is certainly different and that could be one way of breaking into pieces the mathematical community about, which I have to say about, as you know. Now I want to go back to a point which I find challenging. You say that young people are more selectable about peer reviewing than younger people. And it seems to me that this calls for an implementation, and it seems to me that this doesn't fit very well with the idea that people are checking by themselves.

47:30 That is something that doesn't fit together, it seems to me. So, how would you interpret the fact that senior people are more skeptical about appearing in the world? It seems to me we should believe something that perhaps does not appear in the way in which... I mean, there is an evolution there, and it seems to me there could be a kind of evolution. I did not recognize something that I know that trans-mathematicians do when they have to review something, namely that they put the paper aside and they prove the result by themselves, and when they manage to polish, then basically they accept the proof, and it seems to me that this I think it fits better with what the first one was saying, saying that mathematicians were independently checking things out, which is not checking proofs out, it seems to me. So, I am not sure that the interpretation is certainly that they are checking proofs written by another, but they might also be checking the results by themselves. And this is part of the process of analyzing the data that, in my experience, is important. So I was wondering whether you met with that part of the process. Did you, did people comment on this part of the process? Maybe pulling aside, you know, and be doing that... There are only very few comments in this commentary of the, you know, Maybe the senior mathematicians know that there are a lot of professors who take these questions very lightly and give it to the teaching students. But where are they? When you analyze the review process, where are the teaching students? So they are very skeptical if this review process is a real good versus a second good. Maybe this is the simple reason. I don't know.

50:00 Okay, we have a whole lot of questions still. I don't think everyone will get a chance, but I suggest that everyone just ask one question, and that we are brief in the questions. And he has a question. All right, attention. I wonder to what extent people really answered in what they were doing or what they found that somehow was a good mathematician's behavior. How far did you check whether you just got ideology? Because, in a way, this Thurston paper is revealed in a way where mathematicians were wondering what the influence of string theory and all these non-rigorous mathematics had on community standards. But there, the community standards were affected, and nobody came to that conclusion, which I was suggesting not by the committee, which Moritz showed us in his case studies, that in different disciplines, different standards might be applied. So in that sense, and this debate was one about A kind of imagined community of all mathematicians and physicists. It might well be that here in the answers people tell you the ideology because they don't believe that the name of the author doesn't influence if they know it. Of course in many cases they wouldn't know it, but I think if you check the proof of a famous author and find a gap or something, you might behave differently as if you don't know it. It may be, but the theory comes from the famous quality we have here sometimes. No, no, from what people do. Yes, there is more. I think that's an ideology. Maybe it's true. But that's a problem with mathematics. You don't have the motivation for mathematics, for the answers of the mathematicians. Something like that might change. But there might be strategies to avoid this, by other points. That is true. Not simple arguments and to take scales or something like this. But then the question may become longer and longer and we have to wait how many of these mathematicians will answer.

52:30 So this was the first attempt and in this last, in this project study, we achieved very short questions. Do you think it's worth to make such a study with the Pico centers of an international Not before we change the topic of the session, maybe not written down again, because some of you and me are assumed to come out of the room for a long time. So you have to revise the way, because there are few hands that you say were published, others you say are consistent, but where do they come from? You might ask, are published and are consistent? Come on, come on, come on. There's one fair amount of... You can never get a fair question. I've spent five years trying to figure out what you can never get the right answer to. But they can be less bad. Yeah, that's for sure. This one can be less bad. If you didn't understand the point I was making, if you had 20 experiments and you had 5% mathematics, one of them is likely to turn out anyway, and if he's got 10 at the moment, it's a 50-50 chance that one of them is going to turn out. Not that one, but one of them. It's amazing how mathematicians can understand mathematics. I found this out by accident in my own PhD, because I asked about 20 questions, and then I thought, you know, which is significant. Of course I'd asked several of them were, and then I thought, and this was dealing with school as well, and I'd asked questions, and when I interviewed the students, I said to them, I said to one guy, one guy, why did you point yes here? He said, well, because I'd answered no to the previous question.

55:00 We have asked a few of our questionnaires, they regularly come back with patterns and letters of their names spelled out in the return blocks. The ESRC has declared that everybody shall do a year's methodology, so you should be doing a PhD in a day.