We know nothing about the Spring Festival Gala.
Luo Zhenyu once said this in his speech over the New Year.
In 2015, Wechat once sent red envelopes at the Spring Festival Gala, kneeling down for an hour and bowing to the New Year under the impact of the national audience.
Smart Tencent Ali, are three months in advance to prepare for the Spring Festival Gala, but still so awkward.
In 2019, the CCTV Spring Festival Gala red envelope bidding time is very late, only more than a month away from New Year's Eve. The giants feel dangerous, but Baidu raises its hand high: I come! I'll come!
Next, my brother will tell you what happened on New Year's Eve, February 4, 2019.
(Zero) A letter, an engine and a battle from Baidu
The story begins with a letter.
It's no exaggeration to describe a broken kettle sinking a boat.
Hou Zhen Yu
This is really a good beginning for a story.
Over the past few years, Baidu has not had an easy time.
Business is a game of right and wrong, there is always room for manoeuvre. But once the hearts of technicians are broken, Baidu's future will be completely rewritten.
Robin Li knows more than anyone. The greatest generosity to soldiers is to prepare them for a grand battlefield.
Baidu people are eager for a big war to wash away ten years of wind and dust.
(1) Life-and-death battles
Don't think too much, you can guess Baidu's posture of sending red envelopes in Spring Festival Gala.
1. Time: During the late spring, red envelopes were distributed in several rounds.
3. Persons: All Chinese;
4. Action: point by point.
Unexpectedly, it is such a simple four-step process that triggered the next month's bloody stories.
Speaking back to that time.
Wu Yongwei, the architect of Baidu's information flow director who was on vacation in Sanya, also received the news. He kept on returning to his Shanghai research and Development Center for the first time and made the first plane landing in Beijing.
Wu Yong Wei
Zhenyu represents the basic technical support team and Wu Yongwei represents Baidu App's technical team to form a joint operation group. They dare not delay for a second. On that day, they began to prepare for the formation of the red envelope technical team for the Spring Festival Gala.
Chen Xiyang, an 11-year Baidu native, is the God of system performance optimization. His ability is to dismantle an App into small parts like a cattle, and optimize each part carefully, so that the system network resources required by App can be reduced to the theoretical minimum.
Wang Ji, a seven-year Baidu native, is not only a technological bull, but also very familiar with the Baidu login account team.
Song Lei, a 10-year Baidu native, is an expert in network group of system department. His main task is to maintain the stable operation of Baidu network.
Zhang Jiajun, an eight-year Baidu native, the head of the supply chain in the system department, and the new servers Baidu needs on weekdays are all managed by him.
Wait a minute. There's a long list behind.
But after almost a second of hesitation, they became as excited as children.
Doesn't the outside world always say that Baidu can't do in the mobile age? But no matter what others say. Ten years later, we finally don't have to wait any longer. Since the director has given us this opportunity, I would like to prove to everyone whether we can or can not!
He Feng said to everyone.
Soon after all the expert group discussion, He Feng put out the final plan. (The original task list is extremely complicated. Brother Zhong explained it to you in his own words.)
Task 4: Song Lei purchases soft resources such as bandwidth from operators, and builds sufficient IDC and CDN networks. (Time: 2nd week - 4th week.)
Task 6: Chen Xiyang assists the small program team to develop the red envelope wiping program on the day of Spring Festival Gala to ensure that the small program also occupies the least resources. (Time: Fourth week)
Task 7: Wang Ji is responsible for assisting Baidu Passport in all-round consolidation and responding to the login requests coming from the Spring Festival Gala. (Time: First week till Spring Festival Evening.)
Task 8: Notify the major third-party application markets and tell them that there may be a large number of users downloading Baidu App in the Spring Festival Gala, so that they are also ready.
Task 10: On the day of Spring Festival Evening, all the people went all out to resist the 1.3 billion people's traffic tsunami.
Ten missions are here, flag hunting.
At that time, even Hefeng, the planner, could not imagine how arduous efforts Baidu, the three major operators, the global hardware supply chain, several Chinese server manufacturers, outsourcing construction team, computer room operation and maintenance team, tens of thousands of people around the world and their families would make in the next four weeks.
But at this moment, no one can stop it from happening.
This will be a life-and-death battle for my career.
He Feng said.
(2) Space orbiting
The calculation results come out! I expect the flow of Spring Festival Gala to peak at 50 million times per second! A peak of 1 billion times per minute!
Chen Xiyang stayed up all night, holding the latest report to He Feng.
In order to measure traffic flow, Chen Xiyang has studied many news related to Tencent and Ali Red Bag.
After calculation, the cloud computing system supporting these traffic needs to be composed of 100,000 servers.
What does that mean?
Throughout 2018, 9.6 million square kilometers of servers were sold in China, totaling 3 million. Baidu needs to complete one-thirtieth of last year's national sales in a month without any preparation, and complete the whole process of purchasing, production, commissioning and access to Baidu cloud.
He Feng knows that if it is purchased on a temporary basis, it can't be done. Baidu must sell at least 50,000 servers to support the Spring Festival Gala Red Pack program.
You know, Fengchao system is the core of Baidu's advertising revenue. The four-hour suspension of Fengchao means that Baidu's rocket will be forced to stall for four hours at an altitude of 10,000 meters.
When He Feng took the plan to Wang Yue, who was in charge of Fengchao, he was very hearty. He knew that the other party had 10,000 reasons to reject his plan. But Wang Yue only spent ten minutes reading the plan and said three words: No problem!
The remaining 50,000 units will be handed over to the purchasing department. I'll talk about that later. Come back and continue to see Chen Xiyang.
100,000 servers are the limit of the team. But Chen Xi-yang knows that it is still a prerequisite to control the total number of servers to 100,000. The premise is that Baidu App on each user's mobile phone still needs a lot of optimization.
When Chen Xiyang finished his analysis of Baidu App with three brothers, his tears were coming down.
Baidu App will send more than 100 connections to its own server when it starts. These connections come from different business teams in Baidu.
Over the years, Baidu's technology producers have been working hard against the wind.
But for Chen Xiyang at the moment, it means disaster. On the day of Spring Festival Evening, every second, every connection must be multiplied by tens of millions of people. Every App has 100 connections, and every second is billions of connections. BAT must kneel together.
On January 10, Chen Xiyang and the leadership of the headquarters held a meeting. It was unanimously agreed that the number of Baidu App external connections should be cut from 100 to 3.
Because AppStore and major Android stores need audit time, Baidu App must release the latest version two weeks before the Spring Festival Gala, so many links are too late to modify the overall logic behind, and can only be handled by temporary suppression.
There are two flowers, one on each side.
Industry gossip shows that in 2018, Taobao Spring Festival Evening Red Pack, the biggest problem lies in the login system.
Truth be told, because the search business naturally does not need to login operations. Over the years, Baidu's account system construction is weaker than E-commerce Ali and social Tencent.
Before receiving the Spring Festival Gala assignment, Li Ying, the technical director of the account team, was in deep distress because an excellent classmate of his team proposed to leave. However, on the day of receiving the assignment, the former classmate who wanted to leave found Li Ying and said that he had not left.
This means that there will be many people requesting SMS authentication codes.
With limited time, Wang Ji helped the account team to optimize all kinds of accounts overnight, while allowing the team to purchase SMS services from major operators and third-party service providers. For a while, the short message sending ability of the three major operators in all provinces and municipalities was booked out by Baidu.
But even so, it still can not meet the expectations of the Spring Festival Gala. Wang Ji urgently sent Baidu's best engineers to third-party service providers to help them optimize their code and improve their ability to send short messages. Finally, the expected value has been achieved.
The software side is busy. Looking back, there are 50,000 server hardware gaps. All eyes fell on Zhang Jiajun.
Zhang Jia Jun
(3) That crazy night
Beijing Data Center has booked 40,000 servers for delivery before the Spring Festival. By this time, more than 20,000 units had been delivered and 18,000 units remained undelivered.
So Zhang Jiajun's plan came out soon:
1. Urge Beijing Data Center to deliver 18,000 servers in a week.
2. Make every effort to grab 10,000 new servers and put them in Nanjing Data Center within two weeks.
These servers will be integrated into Baidu cloud unified deployment. Zhang Jiajun found that although Baidu Yun usually kept silent, the pursuit of technology for so many years, the more difficult the time, the more it can reflect the value of flash:
The servers designed by Baidu are mostly manufactured in the way of the whole cabinet. That is to say, when the server manufacturer leaves the factory, it is already a big cabinet with 30 servers fixed.
This means that Baidu Cloud does not need to install one server on site like other cloud computing vendors, but can test and install the whole cabinet directly.
This is the whole cabinet installation scene
In the next week, Baidu's wave of partners also reflects the international top professional spirit, a team of trucks neatly merged into the highway, directly to Beijing.
You may not believe that these two big factories together set a world record of 10,000 servers installed in 8 hours.
Beijing is on schedule.
But the most troublesome thing for Zhang Jiajun is how to turn out 10,000 servers in Nanjing Computer Room.
He made an account for me:
The hardware backup cycle in the electronics industry is usually 8-12 weeks. That is to say, you have to place an order with the server manufacturer and parts supplier at least two or three months in advance.
Even if the server manufacturer completely shuts down other activities in hand, the capacity is limited. It usually takes one week from receiving orders to producing them.
It usually takes 3-5 days to drive a truck from a server manufacturer's workshop to a Baidu data center.
It usually takes 1-2 days from server testing and installation to grid debugging.
Zhang Jiajun's task has become to help server manufacturers coordinate the global supply chain and find all the accessories for thousands of other servers from around the world.
If you change to someone else, the task can be declared a failure at this point.
But Zhang Jiajun is not ready to concede defeat.
Over the past few years, he and his colleagues have struggled to understand how the industry chain works, to establish links with major manufacturers, and to build Baidu cloud infrastructure bit by bit. They gambled on their youth and years, but still heard a lot of irony from the outside world. All these years have passed, because they believe that one day all the proofs will come back, and the things that they shouldn't bear will fade away like haze.
At this moment, how can he refuse to admit defeat?
He and his team called global suppliers one by one, bought the latest flight abroad, and went to each factory to check the number of parts allocated. One day, two days, three days, these parts originally planned to be allocated to the United States and Europe, from Israel, the United States, Southeast Asia to turn the course, together to fly to China.
This is a victory for the global supply chain.
Server manufacturer is full of power, all workers abandoned the plan to go home early, returned to work three shifts, servers are continuously produced. Trucks waited at the factory gate, abandoned formation, filled with a car and set off. On the expressway to Nanjing, every 100 kilometers, a truck full of servers was speeding.
In Nanjing Data Center, Baidu staff, computer room operation and maintenance personnel, construction outsourcing team has been waiting for.
Zhang Jiajun coordinated with his headquarters in Beijing and could not get away, so he and Nanjing were on the phone 24 hours a day to direct progress. Server in the computer room, there is also a process of debugging, but any step of error, will lead to a full set of lost. Therefore, Zhang Jiajun's team has made careful plans for every thought of accident.
On the night of January 20, the largest wave of servers arrived in Nanjing, and it was a great success. At this point, the most unlikely accident, but really happened.
The ladder in the engine room went on strike without warning because it could not bear the torment of more than a ton of cabinets. They immediately turned to the passenger ladder, which soon broke down.
Someone took pictures of the emergency repair of the elevator.
Near the Spring Festival, most elevator maintenance personnel go home for the Spring Festival. They are very nervous and need daybreak to arrive. But Baidu's classmates know that they can't wait just for one night.
In the dark night, all the people on the scene, with their hands and shoulders to carry more than a hundred kilograms of servers, step by step from the stairs up.
At that time, a classmate recorded the scene by video:
Human flesh carries the server upstairs
Some classmates'hands were cut and blood beads oozed out. My colleague wanted to change it. He just shook his head and said it was okay. Tears were seen in the corners of his eyes.
There are many cabinet servers with more than one ton on site, which can not be moved upstairs by human resources at all. The students of the field command even called a crane to break open the windows of the engine room and bring it in.
Zhang Jiajun watched the hook of the crane shaking in the wind and was startled into a cold sweat. Fearing that his classmates would be hurt, he rejected the plan in words. But the person in charge of the scene repeatedly pleaded: Let's try! Server to go upstairs!!
Finally, the on-site maintenance team came up with a way to remove the elevator parts from the next building. The elevator finally started slowly. By then, it was already two o'clock in the middle of the night.
At 8 o'clock in the morning of January 21, 2019, the students in charge of server software debugging arrived on schedule. All the servers were standing quietly in the computer room. They were so silent, as if nothing had happened last night.
(4) Nuclear warheads in place
On January 17, Baidu convened a board of directors.
In fact, those days were the worst time for all teams.
Zhenyu told me that just this month, Baidu's IDC added more bandwidth resources than the past 20 years combined, and CDN added half of the resources in 2018.
According to Song Lei, in about two weeks, 69 engineers flew more than 70,000 kilometers. Beishengguang's remaining bandwidth resources are almost empty by Baidu.
Song Lei and his friend Cola Jun
Two engineers in the network team are husband and wife. One is responsible for network construction and the other is responsible for network testing. One person just built it, went home to look after the children, and another person came to test it. Sometimes the construction engineer would work with the test engineer, and the two men would have to throw their children to the old man and not go home overnight.
Song Lei himself, from January 6 to the first day of New Year's Day, slept for five hours the longest night and only one hour the shortest.
A classmate lay sleeping on his desk and was photographed mercilessly
It's January 26, and it's all the time. There are still eight days to go before the Spring Festival Gala.
On the other hand, Chen Xiyang succeeded in cutting down the number of Baidu App external connections from 100 to 3, and Wang Ji increased the landing capacity of Baidu App from 1500 people per second to 150,000 people per second.
1. Every time the host broadcasts the red envelope, how many seconds in advance does Baidu prepare for it? It's as precise as a rocket launch.
2. Once encountering an unexpected situation, which subsystem should make how to adjust, according to the different degree of unexpected, the adjustment efforts are also different. This set of plans involves thousands of unexpected situations. Once the condition is triggered, the students in the command can start the corresponding changes by clicking a button.
For Baidu engineers, every plan in these hundreds of pages of battle plan is not imaginary. When they write every word, they can recall their own operation and maintenance experience in Baidu for more than ten years. This is a thick book. Where is the battle plan, it is clearly a technical life of Baidu engineer.
The whole hall of the K2 building of Baidu Science Park has been transformed into a battle command room with a core command room in the center. The next 20 cabins are group battle rooms including Baidu App, Dasou, Shake, Account System, BFE Platform, Network, System Monitoring, IT Team, Red Pack Core System and so on. The joint command and combat system is composed of an existing control room on the ground floor.
Core Command Room
Here's a slight interruption.
This creates an open space: inevitably someone speaks a dialect, or refreshes the waterfall flow in the wrong direction, which brings unpredictable tremendous pressure to the backstage AI system.
This will lead to test time, it is very difficult to simulate the real scene of the Spring Festival Gala.
Necessary Step: Baiyang Beyond
Although there are several twists and turns in the middle, in the last two joint tests, the whole Baidu Spring Festival Gala Red Pack system has withstood the test of 50 million visits per second. Everyone's been hanging around for a month, which is just a little lower. Wu Yongwei, who has always been rigorous, told the team members that my confidence index now is 85-90 points.
Just as everyone was busy, two strange things happened:
In those weeks, Song Lei added a habit of taking pictures of himself working overtime at two o'clock in the middle of the night and sunning them in the circle of friends. At that time, we still wondered why the old Song was so low-key, but he had to work overtime every day.
Wang Ji, who came from Shanghai to support him, disappeared mysteriously for a day on January 24. Where on earth did he go?
These little gossips, nobody had time to explore secrets at that time. Because the Spring Festival Gala is near.
(5) New Year's Eve
February 4, New Year's Eve.
Since zero o'clock on that day, classmates have been on duty in the Department of Operations. At eight o'clock in the morning, the whole team was ready and the war was on the horizon.
According to the design, at 11:00 a.m. on New Year's Eve, Baidu will push a small red envelope activity to users as a warm-up, so that real users can participate, so as to carry out a wave of solid final testing of the system.
At 11:00, the warm-up activity began, and the background data rose sharply. Baidu App instantaneous access peaked at 880,000 times per second. That's tens of times the historical peak of Baidu App. But Hefeng knows that it's just a drizzle. They designed 50 million times per second for the Spring Festival Gala.
At 8 p.m., the Spring Festival Gala starts on time.
The phoenix nest system slowly extinguishes and the red envelope system takes over the engine driving. All system gears bite and rush out like a race car in front of the starting line.
The students in the battle room seem to have entered another world. Throughout the building, you can hear a pin drop on the ground.
A minute later, in the public opinion monitoring group of the headquarters, someone suddenly threw in a picture:
Apple's AppStore has been crushed by netizens and can't be opened.
The battle group immediately gave the data, and it is estimated that 2 million to 3 million people in the country can not download Baidu App, which will bring considerable losses. Another group of data shows that people who can't download App are flocking to Baidu search again, trying to download Baidu App by keyword search in mobile browsers.
He Feng assessed the occupancy of Baidu CDN, immediately ordered that the link point to Baidu's own download interface, so that we do not need to use the third-party market, but directly download Baidu App from Baidu's home. In this way, the download peak directly hit Baidu's own network, bit by bit, millions of people installed Baidu App.
All this was in Wang Ji's eyes.
His first reaction was: our factory is really awesome. His second reaction is: after these 3 million people downloaded App, they must log in. The login system I'm responsible for seems to be hanging up soon...
Fortunately, there is no danger, we decentralized download App, also decentralized login App, did not bring fatal blow to the login system.
As the Spring Festival Gala progresses, the traffic to Baidu App is bigger and bigger, approaching the expected peak.
Baidu's accumulated data access throughout the country in the past 20 years smoothly carried the data peak of the first three rounds of red envelopes, leaving only the last time before the sound of 0 o'clock. It is predicted that this will be the largest wave.
Twenty-three minutes before the last red envelope arrived, He Feng suddenly received the news from the two students stationed at the Spring Festival Gala: according to the calculation, the Spring Festival Gala was four minutes later than expected.
He Feng had a chuck in his heart.
This will result in Baidu App red envelope snatching time earlier than the host broadcast time. That is to say, the user who comes in only after hearing the anchorman broadcasting will probably find that the red envelope has been snatched up by the user who starts on time...
He Feng needs to make a decision: to keep the red envelope open as planned, or to postpone the red envelope opening time on App according to the schedule of Spring Festival Gala.
It takes at least five minutes to deploy such a large-scale adjustment to 100,000 servers.
Everyone looked at Hefeng. He Feng stared at the screen and pressed his hands. He said to everyone, "Wait a minute."
Ten minutes before the red envelope was opened, He Feng judged that CCTV should not be able to recover the time. He ordered that 100,000 servers be immediately instructed to open the red envelope system four minutes later.
God's favor, finally Baidu App red envelope opening time, and the host announced that the red envelope robbery time closely matched.
In an instant, hundreds of millions of mobile phone screens display Baidu App's red-envelope interface, and huge data surge to data rooms in Beijing and Nanjing. At that moment, more than 1,000 colleagues in Baidu Mansion, 100 colleagues scattered in the computer rooms of Baidu, engineers of more than 100 server manufacturers waiting in the computer room with spare parts, more than 1,000 colleagues left in the computer rooms of the three major operators in order to protect the network smoothness, those dreamers of China's telecommunications industry and the Internet industry, built up people with their own efforts. The broadest information path in the history of the class.
The night was silent and fireworks were everywhere on the earth.
This ancient nation has entered a new year.
(6) Those trifles
The red envelope system, which burned for four and a half hours, gradually went out.
The phoenix nest system started slowly after four and a half hours.
With the hissing of 100,000 servers, Baidu's rocket completed its second thrilling orbital change in space.
In the headquarters, cheers thundered.
Unfortunately, such a magnificent project has mobilized the global supply chain system, mobilized the network bandwidth of China, and mobilized the technical resources of Baidu. In the middle, if Baidu system has any technical link stuck, Baidu cloud's server has any screw loose, Baidu engineer, business purchasing team has any one person out of the chain, the whole Chinese people will face a completely different outcome.
Baidu is undoubtedly lucky, but in the world, only brave people are qualified to talk about luck.
I heard a lot of Baidu students recall that moment, they described, as in the "Shaolin Football" scene:
After suffering, all the martial arts of Shaolin brothers and brothers were awakened. Zhou Xingchi knelt on the ground and said: Welcome to your home!
Until the end of the first month, when I came to Baidu Science Park to interview the story of the red envelope in the Spring Festival Gala, I could still feel the joyful atmosphere in the building.
Oh, yes, there are some small things that I forgot to explain.
And curious people finally got to know the reasons for the mysterious events of the past few days:
Song Lei shyly said that the reason why he sends out a circle of friends at 2:00 a.m. every day is that he is afraid that his wife thinks too much and exposes the circle of friends to prove his innocence.
On the day Wang Ji disappeared, he flew back to Shanghai to celebrate his son's two-year-old birthday. Because last year, on his first birthday, he worked overtime in the field and couldn't accompany him. This time the son shouted for his father, and he flew back all night for one day. The next day, after his son's birthday, he went to bed with him at 8 p.m. to make sure that he was asleep. He rushed out and flew back to Beijing at 10 p.m.
Eleven years ago, I joined Baidu as soon as I graduated. Baidu gave me all my technical accumulation. I regard Baidu as home. Previously, Baidu encountered many problems, I feel more uncomfortable than the outside people, just like their children, hate iron is not steel.
I remember that in 2016, when Baidu was most difficult, our team went to Hangzhou to build a delegation. Hangzhou was very beautiful in autumn. One of my classmates who lives near Hangzhou resigned and went to Hangzhou.
I know that everyone has the right to make their own choices. From 2016 to the present, almost every week, headhunters call me, but I always hang up directly. I think there are always people who want to make Baidu better, and I should be one of those people.
He said so quietly.
Hearing this, I suddenly understand that at the moment of the success of the red envelope in the Spring Festival Gala, the tears in the eyes of Baidu technicians are not consolation for the hardship of the past month, but for themselves ten years ago, the teenager who does not understand the customs but does not care about the West and east.
Chen Xi Yang
They endured loneliness quietly for a long time, but this time they did their best like madmen, with all their youth and blood. They may not want to win, they just don't want to lose.
Some people say that life is a battle, the end is either a long-standing advertisement, or an eternal regret.
But I am more convinced that time is a long test, no one will force you to hand in the paper.
You can spend your whole life giving your answers slowly.