Planning Editor Natalie
By Jake Edge
AI frontline introduction: In the past few years, the adoption of Python 3 has increased significantly, but it still has a long way to go. Large companies using Python tend to run a lot of Python 2.7 code on their infrastructure, and Facebook is no exception. At this year's PyCon 2018 conference, Facebook product engineer Jason Fried described the company's entire process of Python 3 from being almost unattended to becoming the company's mainstream Python version for the past four years, and also showed Fried as an engineer. Perseverance.
Jason Fried, a product engineer at Facebook, played an important role in helping the company achieve this goal, and he discussed some ideas on how to solve the Python version migration.
Fried entered Facebook in 2011 and soon discovered that he needed to learn Python himself, because on Facebook, Python code is easier to pass code review. Later, he found himself a major driver of Facebook's adoption of Python 3. He said that he had never planned anything but that Python was used much more naturally.
Jason Fried first appeared in the Python community and was very active. He was often the first to stand up and answer questions. Subsequently, he became fame (or rather "infamous") with Facebook as a supporter of Python, because when he saw problems in the Python code, he would immediately modify it without permission. This works for Facebook because there is no real top-down control mechanism. Everyone has the right to make changes to a code change, just as you have the right to make code changes. Over time, he established prestige within Facebook's internal Python community, which has greatly contributed to his successful Facebook-led migration to Python in the future.
This is a complete timeline for Python 3 on Facebook from ignorance to dominance, mentioned in Fried's speech. It can be seen that this process took nearly five years, which is not easy.
2013 (basic support → negative emotions → hope to present)
Python 3 can never appear on Facebook
Python 3's landing process on Facebook was very difficult. At the beginning, it was internally negated. Even Fried thought that it was impossible to appear on Facebook. Until now, the adoption rate exceeded 55%. The whole process is very bumpy.
He said that it will take quite a while to change things like the Python version on "Facebook scale", and it needs to use a lot of "diplomatic" methods. He described how he and several engineers used his free time to make Python 3 a major version of Facebook without any power.
In 2013, Facebook intended to begin initial support for Python 3 because they needed to add Python 3 support to the build system. However, because the Facebook library does not support Python 3, it is not possible to add Python3 to the build system. And if the build system does not support Python 3, the Facebook library cannot support Python 3. This is just like the contradictory military regulations described in "Catch 22". Although Python 3 is "available", it does not receive any support in the Facebook environment.
In addition, in 2013, Facebook has a lot of negative feelings about Python 3. In general, they think that the company's programming language will always stay in Python 2.7. There are also suggestions to completely switch to another language. Fried also said that (in the internal community) Python 3 will never appear on Facebook. Only one person challenged him and advised him to do something to change the situation. Although he ignored this suggestion at the time, the idea remained in his mind.
Hope to present
In 2013, things turned around. In January of that year, the "linter" tool that Facebook was using at the time needed toFutureImport print_function, division, absolute_imports, and unicode_literals to extend the useful life of the Python 2 code base. They import these packages at any linter prompt, which makes it easier to convert modules to Python 3.
The Apache Thrift framework for serialization and remote procedure calls is on Facebook"ubiquitous”. Because it only supports Python 2, it is the biggest obstacle. However, a survey of Thrift's new features by the Facebook Thrift team revealed that developers generally want to be able to add Python 3 support. Fried voted in favor, but it wasn't a follower. He thinks the Python 2 interface needs to be refactored because it looks like Java.
When he saw Guido van Rossum's Yelp in San Francisco talking about something called "Tulip" (which eventually became the asyncio module), his ideas began to change. He has been a fan of Python asynchronous programming, but has been fragmented because of differences between frameworks such as Twisted, gevent. Tulip makes interoperability between asynchronous I/O operations. Before the end of that speech, he communicated with the Facebook Thrift team, stating that Thrift should support Tulip directly, instead of waiting for Twisted, gevent, and other frameworks to migrate to Python 3. A few days later, the Thrift team released a roadmap with support for Python 3 and Tulip.
The Thrift team introduced these two new features in early 2014, but there has been little movement for the next six months. The user did not respond to this, in fact they did not care about, or even do not know that these changes have taken place. Fried also cites an example of China's building but no one to illustrate it. It's really ridiculous.
2014 (changing culture & rarr; starting from scratch & rarr; mandatory)
In August 2014, he began to rewrite a service and planned to use gevent and Python 2, but he later realized that if he did so, it would be obsolete when the project was completed. In order to change, someone needs to be the first person to make a change. To promote the use of Python 3 in Facebook, that person is none other than Fried.
So he started his project with Python 3. As you can imagine, what he faced was a "broken" situation. At that time, Facebook had no one using Python 3, the build system did not support his code, and all third-party packages were only for Python 2. After he fixed all the problems, after the code was compiled, there was a problem at runtime.
In order for the code to work properly, he must fix all the problems. He rebuilt hundreds of third-party packages so that they can support both versions of Python simultaneously, and he must make all internal libraries compatible with Python 2 and Python 3. However, every day people submit changes to Python 2 to his dependencies. He needs to keep fixing problems and is tired of it. One solution is to enforce Python 3 compliance within the organization, but this is simply not possible on Facebook. However, if you act as if there is some power, people will gradually believe that you really have this power.
He used a lot of relationships to add Pyflakes (a lint tool) to the build process. He can prove that adding it makes sense because although PEP 8 is already in place, Pyflakes can solve other additional code quality issues. In addition, Pyflakes has almost no false positives, so it won't provoke developers. He made some settings that allowed Pyflakes to scan all the code that needed to be reviewed, first Python 2 and then Python 3. This helped to extend Python 3 compatibility to all developers, not just him, which made his project progress.
At the beginning, he must spend a lot of time explaining to people that "linter is not wrong" and that it is worthwhile for the code to run on Python 3. If developers begin to feel that migrating to Python 3 is a difficult task, they will return to "Let's stay in Python 2" forever. He wants to make sure that developers can run code on Python 3 smoothly.
Despite some of the difficulties, there has been little or no progress in Facebook's expansion of the Python 3 site. He joined the team that trained Python programming for new Facebook employees. He wants compatible code to be used only for legacy projects, while new projects should be developed with Python 3.
In 2015, he revised the Python training content for new employees, stating that Facebook will one day shift to Python 3, and that writing only Python 2 code is meaningless because it has to be rewritten in the future. He teaches new employees that all the code should be consistent with the Facebook infrastructure and build system, and if not, they should submit errors or try to fix themselves. In this way, new employees begin to use Python 3 at work, which is the beginning of progress. “The weird thing is that things are happening.”
In January 2015, he finally delivered his project. It took him half a year to tell people how good it is and why they should use Python 3 as much as possible. Over the past year, many of the allies who have worked on Facebook to promote Python 3 have made a name for themselves in the company.
2016 (Python 3 becomes the default programming language)
In 2016, he found that although the adoption of Python 3 has been growing slowly, it has seen steady growth. People mentioned it at the conference, and he often heard new projects using it. Even if Python 3 is not the default setting, the project will choose to use it. Facebook's view of Python 3 has changed. In May 2016, Fried expressed his intention to switch the build system to use Python 3 by default. His proposal was almost absolutely supported. A few days later, he completed the switch and did not have any adverse effects after the switch.
Fried said that in 2016, only ten people promoted the Python 3 project in Facebook. Three of them are the main promoters, and the flow of personnel is constant. Many people who work on this project are part-time.
At the end of 2016, a project team published an article describing the results of switching to Python 3. Developers only need to make some fixes when switching from Python 2 to Python 3, running code up to 40% faster and using only half the memory. This broke a rumor that Fried heard before: Python 3 is slower than Python 2. Earlier versions of Python 3 may be like this, but certainly not now, he said.
2017 (Instagram migration)
Good things happen
At the beginning of 2017, Facebook experienced the glory of Python 3 migration because Instagram completed the Python 3 migration. The original version of Python was not terribly upscaled, but brought new features that were available. Facebook developers are now starting to use new static types or use asyncio to retrofit older services. “Python has started to become very interesting on Facebook again“.
The problem now is that everyone is asking when to stop supporting Python 2. When the Python 2 support library or module comes back, developers are often asked if they can upgrade directly to Python 3. And a few years ago, the situation was exactly the opposite. "Oh, the world is beautiful! ”
2018 (Python 3 accounts for more than 55%)
He shows a graph of Facebook's Python service portal endpoints over time, starting from the third quarter of 2015, when there were only four Python 3 service portal endpoints. By the middle of 2016, when switching to the default use of Python 3, Facebook already had 4% of service entry endpoints using Python 3. In March 2018, this figure exceeded 50%. In mid-May, when he delivered a speech, the proportion of Facebook service portal endpoints running Python 3 had reached 55%. On Facebook, code that can only be run on Python 2 is now in an awkward position, Fried said.
Near the end of the speech, he outlined the speech. In general, his suggestions include:
What you have to do is to innovate, make changes, and the results will naturally come;
You must guide developers through “ to personally let people see the changes you want;
You should also seek the help of others, not alone;
In addition, it is important to train new employees to achieve your future goals.
Collect the required data;
Enjoy the results, write some "great things" with Python 3.
Finally, he also answered some questions raised by the audience. Someone asked how to achieve the goals stated in the speech in a traditional, well-defined organization. Fried believes that this may actually be easier because you don't need to convince thousands of developers that you just need to make management aware of the benefits of this matter. This can be difficult if you are in a culturally conservative organization, but focusing on code quality improvements may help. Another question is about the overall code, not multiple entry points. For this question, Fried suggests looking at the Instagram keynote on PyCon 2017 (see the beginning of the article).
The entire speech has benefited a lot, including Fried's emphasis on advocates and leaders, and the importance of perseverance in a project.