Let's think about a new culture of science.

Academic production before internet

Today, science is produced by the following basic infrastructure. This simplistic picture depicts how humans in a lab interact with the nodes making up the infrastructure. Humans basically use computational resources to analyze data by writing code. The input to these nodes overall is extremely sparse. That is we generally do not have other peoples' data available, we generally cook our own code specifically for our own data, we generally use our own computational resources.

Unit infrastructure for scientific production

This unit is the fundament for academia. Research is carried out by the same infrastructure that is simply replicated across geographies and time. Of course, there could be labs collaborating with each other, of course we could be using an external grid engine to run our tasks, of course we might download a toolbox to run analyses. These are all connections that are not shown in this picture. 

But the emphasis here is that we spent most of our time to reinvent the wheel by

-writing the same piece of code that many people had done it in the past, 

-collecting yet a new dataset instead of generating a new hypothesis compatible with available datasets, 

-buying large computers that could be used by other people

-hiring system administrators that are doing exactly the same work as in another lab

the list is long...

Each lab in this culture becomes a specialized idiosyncratic creature with its own way of doing things. Politically this implies committing in long-term fixed-costs to maintain an academic infrastructure that is short-sighted and benefits mainly to the labs short-term agenda.  Academia mainly benefits from the contributions of labs in the form of publications, which is considered as the unique currency in academic reward system. What is the impact of this system on the society? In the light of current replication crises in science, it is hard to be optimistic.

Infrastructure for academic work. Culture of mine.

This type of infrastructure organization has mainly historical reasons. This model is archaic, and has been a good model for the pre-internet era where people and systems were connected sparsely with each other, where it made sense to travel to a conference and meet other people.

A new way of doing science at the age of cloud-based systems

We have to rethink about how to place boxes shown in the previous pictures, how to set novel incentive mechanisms, and how to organize the work flow across scientists and nodes. Let's talk about this simple picture.

A novel infrastructure for academic work. Culture of sharing.

Outsourcing the storage and compute resources to a cloud service (e.g. AWS, GDC or some supranational public cloud service yet to be put in place) are for the benefit of the society in terms of reducing overall costs. 

However the main point here is not about outsourcing storage and compute resources. The real reason for this move is for making datasets accessible to other scientists. And in the long-term this actually means making data to be publicly available to all citizens.

The only thing that is specific to a given lab is the data that is collected there. That's what labs should do: collect data. Most importantly data must to be stored according to strict standardization. That is to each data set, a map has to be associated, that will help people on how to navigate this data set. Furthermore, every dataset should be stored with a minimal code that ensure basic access to data. Also, most often datasets spans multiple modalities. For example, my fMRI datasets are typically bundled together with pupil recordings and heart-beat recordings. We therefore need not only a standardization for storage of specific kind of datasets, but we also need a way to create dataset-bundles that represents an experiment in a flexible manner. A principled way to bundle standardized datasets. Let's call this step 1.

The other thing that labs do is to write code to process their data. To my opinion this is where the biggest challenge is located, namely on finding a system where people can collaborate and create something together. Assuming that the step 1 is solved, the code that is written will also be publicly available. Therefore, code that is written will be directly connected to a dataset type. 

For example, if I am trying to detect peaks in a more or less periodical physiological recording, I will not start looking for literature, find someone's algorithm, implement my version of it. I will simply search for code that is compatible with this type of data, browse among alternative codes, read comments to figure out strengths and weaknesses, consider ratings and incorporate that code to my pipeline.

Basically putting up an analysis will be about creating a pipeline using previously coded nodes or coding new nodes when the analysis has not been previously carried out. When something doesn't work as expected, code needs to be improved via collaboration. Writing good quality code will be one great novel incentive for scientists.

Another challenge is to find a way how to fund this novel system. This is certainly beyond the capacity of a single start-up. This is also beyond the scope of a single lab or institute. I also don't think today's national states are visionary enough to take such moves. To my opinion this could only be established by some tech giants who have the know-how required to solve all these problems.

What is so bad and great in academia?

When you are a scientist working in public service i.e. in academia, a major and constant question you ask will be about identifying deep-rooted problems in academia. While there are actually many bad aspects of academic working conditions, there are also great things about it. This post tries to give an objective pros vs. cons perspective to this question.

I started a list of bad and ugly things that are constantly deteriorating academic life. A list of great things follows below...

• Pressure for your own future
  
  Well, no surprise here. If your study fails to find an effect, you are the first one that is affected. You wasted two years of your life and didn't make any progress, your boss lost interest in your project. Depending on the workplace culture you might have absorbed such risks if your project list was diverse enough, but generally the rule is that one person = one project. There is no mechanism currently that absorbs this type of risks. Result: Scientists are pressured to find something, creating overall a bias for falsely finding positive results.

• Backward publication system
  
  
  We publish more or less as Fisher or even Newton. Well we do not really send a post mail but an electronic mail to the editorial office, that is true. But besides that the publication system has not seen a major improvement incorporating anything positive from the era of internet. This model is hackable by short-term benefit seekers and introduces biases in findings. A. Gelman discusses here how changing the publication system by shifting towards an open-post publication reviewing system can create novel incentives for the community.

• Lack of crowd-based knowledge systems 
  

  

  Ratings: ★★★★☆
  412 scientists rated
  this broom with 4.12 stars.
  It is a great broom!

  
  We scientists read many many papers. But we somehow cannot give any opinions on them. Whereas a person selling a simplest plastic broom could receive harsh comments on Amazon, a person who is writing a bad paper in an high impact journal can easily get away with it. Why can we not simply rate papers online, why can't we create a crowd-sourced reputation system for papers that is fair and transparent in the same time?

• Culture of mine
  
  As soon as you are born as a scientist, say when you start you master thesis, you will be assigned to one single project. When you grow up and get your own grant you will do the same, you will assign one project to one person. This makes you live in a bubble and cuts you completely from all sort of sharing tools and mindset that are simply the standard in industry. Unless you dedicate your own time you will never learn great practices of code sharing, writing a code for others, encapsulating your analysis as a toolbox. This is because you work alone for long long periods of time.

• Pressure for publishing
  
  Publishing is a key activity in academia. The problem starts as soon as your qualities are judged solely by your publication track record. What about teaching, supervision, peer-reviewing, code sharing, diversity of your publication track record? Nope. At the end of the day, only the impact factor of the journals you published counts. If you started your career in a great lab, but has never published in high-impact journals, you have already started with one leg missing. Combined with the "culture of mine", this opens the way to authorship disputes, that are everywhere.

• Pressure for short-term thinking 
  
  Most of the position in academia, are short-term contracts. Great masses are hired by a few professional elites. And these elites are equally free to either plan with you and invest in your career, or to exploit you like a vampire until your contract expires. There is no mechanism that evaluates supervisors in terms of the success of their students.

• Lack of transparency in evaluation of your work 
  

  

  Reviewing papers anonymously.

  
  It takes on average few years to work on a project and finalize it as a manuscript. The evaluation takes 45 minutes per reviewer that are free too evaluate your manuscript as a crocodile or a sweet hedgehog. Accountability of reviewers is not in the equation, same for inter-reviewer reliability.
  
  

• Predatory seniors
  
  Academia is a social service, the person who has the title of professor is a civil servant. However, there are no mechanisms that evaluate professors on their performance with this respect. A question like "What did you do to improve scientific practices last year and make the system more efficient ?" is missing.

• Lack of recognition
  
  The fact that you can spend a lot of time working in academia and collect lots of expertise and experience, doesn't entitle you with anything significant. As far as I can talk for Germany, you are just an employee, not a hair-dresser, not a pharmacist, you are just an employee. Why don't we have a profession called "scientist"? This is mainly due to the lack of long-term contracts. And I believe very strongly that third party funding is damaging the academic sphere in favor of few strong elites and at the expense talented young scientists.

An article that focuses on only the bad sides cannot be useful for anything. So let's actually talk about what is great in academia.

• Relative Freedom
  
  Not being constrained by a final product that needs to do something precise gives us great freedom in the way we work our way through something. As a professor, one reaches the peak of this freedom and can work on any topic at any time. It is just a great thing to be able to start a project any time on any topic without somebody telling "you have not published enough on this topic yet!".

• Great colleagues
  
  Working with alike-minded colleagues around you, who are curious and have low thresholds for brainstorming on random topics is a great positive thing. Having a constant hunger for curiosity as a social norm is certainly a positive thing.

• Publishing a great paper
  
  Dedicating your efforts on something and walking step by step on that direction is a great source of happiness. And additionally, crowning your final work with a great publication is a priceless reward. It is something you can show your grand (mo/fa)ther and evoke interest on totally random people.

• Learning doing new things, creativity
  
  Being able to do things that the overwhelming majority of human population cannot is a great feeling. You have spent all your day doing this weird analysis and it turned out to be completely useless, but well you were unique and used your creativity. To the extent working in academia nourishes this craftsmanship it is an extremely pleasurable occupation.

• Flexibility in working
  
  As an academic we are most of the time free on where and when we want to work. You can wake up at 11 and work until midnight or take the opposite approach, it is completely normal to not expect people to conform the regular working class habits.

This is a list I will constantly update and improve. However, this is a great point in time to hear about what people think about the good and bad sides of academia.

The culture of "mine" in science at the age of cloud-based analysis systems*

Scientists needs computing power and storage space for their data sets. For scientific institutions, this translates onto long-term fixed-costs that are relatively high. Resources required for buying hardware, keeping network infrastructure, paying system administrators to take care of these masses of electronics is a considerable overhead. As a result, public scientific institutes spend lots of money and human resources to create and maintain infrastructures for storing and analyzing scientific data sets.

Yet, one scientific institute is pretty much the exact replica of another one when it comes to hardware demands. That is, resources that are needed in one place should in theory be very similar to another place. Therefore, instead of investing money for system administrators, storage and computational resources, scientific institutes may actually lease these services from cloud-based infrastructures with more flexible pricing opportunities and lack of overhead. Replacing your system administrator with two PhD students is an appealing idea after all.

There is actually nothing illuminating in this view because this has been actually happening already since more than 10 years in the corporate world. Many hosting companies offers as also VNC based system to connect to their servers and use software on powerful machines. Beyond simple hosting companies, Google Cloud Computing and Amazon AWS making the transformation real by integrating all sort of compute, storage, parallelization tools and selling it as a service.

Where are we in neuroscience? Some important milestones are becoming finally a reality in natural sciences, I think that the point of no return is also being slowly reached for neuroscience. I believe this because standardization procedures on how to store and share datasets is becoming more and more mainstream, and this shift has the potential to change day-to-day scientific enterprise radically. For example, Open Neuro is one such platform, where you can upload your brain imaging dataset using the BIDS format, and let analyses run on these servers. I think this is just start of a big scale transformation on how we do science.

Here is how I think how:

(1) Scientific publication

The way we publish our reports didn't change probably since the times of Fisher or even Newton. The world today is a very different place, but many of the novel tools that have been invented in the internet-based communication era have not been incorporated into the way how we conduct science today. OK, instead of sending a manuscript to the editor's office via post, we are today using emails, fine.

For example, the scientific reviewing system did not incorporate crowd-sourcing mechanisms to evaluate the quality of scientific papers. The decision of whether a manuscript or research proposal is worth being published stays largely within the hands of few not-randomly selected referees and an editor. The process is opaque, prone to biases and has no means to stop formation of small-world cartels that mutually benefit from positive biases.

The re-distribution of reputation is not based on metrics that reflect the long-term value of person for science in general. In the best case, reputation is equivalent to your h-index, which is heavily biased by the random success of your publication track, not how good a scientist your are. Metrics that ensures long-term advancements of science are typically not included. For example, we lack a metric that judges a professor based on the number of students that became also professors in the last 5-10 years. The infrastructure to achieve a better and more democratic system is in place since more than a decade. I believe this change will come faster with cloud-based systems decreasing the cost for storage and computational resources.

In the very near future, I believe any serious publication will also need to contain the related datasets, the analysis pipeline and make it publicly available to all scientific community (but also other citizens). This is already happening, and many journals let you agree with their terms of sharing data promptly when requested. However, the definition of "prompt" is also very subjective. For example, you may want to read this twit-storm to see a recent example. Even if it was obligatory to upload the dataset, the re-evaluation of the data is not within the responsibilities of the referees. This means that modifying an existing system incrementally to make it more and more suitable for the current demands of scientific democratization is not enough, we need a radically different way of publishing science.

When the data is stored and analysis ran in a cloud-based system, there will be no more excuses for reviewers for not being involved in the data analysis, as the time it will take for them to have a closer look on the data and the analysis pipelines will be insignificant. Therefore, I believe that any serious publications will take the concerted efforts of, on the one hand authors who designed the experiment, collected data and wrote the initial draft of the paper, and on the other, reviewers who will be required to contribute in the data analysis using infra-structures provided by the cloud-based storage and computational infrastructures. There will possibly be not much difference between collaborators of today and reviewers of tomorrow.

(2) Cloud-based analysis

Most of published reports use similar methods, which are re-invented again and again by generations of PhD and postdoc crew, which is a complete waste of time and resources. I believe actually there could possibly not be a more inefficient system than today's science. A large company would not be able to function like this.

Once we start talking about cloud-based storage and analysis pipelines, it will also be possible to run these analyses automatically on a server. You will need to tick the checkbox for this or that analysis and receive the results as an email in the form of a presentation or a web page (example) to click/browse around. This is of course an over simplification, but what I would like to say is that scientists will spend more time on (1) standardizing their datasets to be able to run analyses on the cloud-based system and (2) making analysis pipelines that are compatible with standardized datasets. Therefore, many scientists will use this time to record more data.

(3) End of culture of "mine"

One of the most intriguing anthropological traits of the daily scientific enterprise, is what I call the culture of "mine". This is not something that is somewhere out there, it is right inside our offices. By this I mean the way how students, PhDs, postdocs and professors (the whole crew basically) are closed to the idea of sharing and opening their projects to external influences. Most often if not always, a project is assigned to a single person in the lab, and this person is expected to run this project until the end. Because the person believes that it is her/his project, he/she can control the monopoly together with his/her boss on how this project has to run and adjust the level of external factors (politics). This results in a very conservative set of interactions between people, as any request of help, or any communication can be seen as a contribution to the project. The culture of mine, will of course be there and start the appropriate set of behaviors to not let this happen. Unfortunately, there are countless examples of authorship disputes which appear exactly from this type of culture.

Once the opportunity to upload your dataset and run your analysis in a cloud-based system is within the reach, there will be no reason to not open your data and let other people analyze it in ways different than what you have actually thought would be most appropriate. In a crow-sourced science, you will own your data, but will actually allow other people to look into it. Pretty much the same way, when people are allowed to look at you when you are walking in the street. The constructive discussions that follows during this process belongs to all parties and can be moderated by the person who created the dataset. I believe there will be a shift in the way how people conceptualize the way how they own projects and data, replacing culture of mine with crowd-sourced intelligence.

I found this article from Jeremy Freeman, entitled "Open source tools for large-scale neuroscience" which made me super happy as it expresses many of the thoughts I scratched on this post in a systematic and professional manner.

*This article has a bias from the perspective of a neuroscientist.