Tcp on ZRJ | 学习笔记

RabbitMQ 消息队列的 Nginx 反向代理

Wed, 14 Dec 2016 15:18:14 +0800

对于 http 协议，大家都知道可以使用 nginx 进行反向代理，那么，对于 RabbitMQ 等有状态的长连接呢？配是能配，但是主要会关心长连接超时的问题。

先来看看配置是怎么配的

$ cat  nginx.conf
#user  nobody;
worker_processes  1;

#error_log  /var/logs/error.log;
#error_log  /var/logs/error.log  notice;
error_log  /var/logs/error.log  info;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

stream {
    upstream rabbitmq{
        server xxxxxx:5672;
    }
    server {
        listen 5672;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass rabbitmq;

    }  
}

可以看到这里用到一个叫 stream 的东西，这个东西是 nginx 新出用来搞 tcp 代理的， http://nglua.com/reads/15.html 这里有对他的说明和源码分析

nginx因为在http服务的优异表现被大众认可，但是它不仅仅是一个http服务器，也是mail代理服务器。现在这个家庭加入了新的成员tcp。其实它在nginx官网的说辞里叫stream，之所以呈现在大家面前是tcp的原因，我猜测是这样的： a：它解决了代理需求，而这需求就是tcp代理，但是它的源码里命名是stream，可以说看不到tcp。 b：确实有第三方模块（ngx_tcp_proxy_module）做了同样的事，这个优秀的模块早好些时间，有兴趣的可以去github上看看。 c：tcp比stream容易上口，这好像有点废话，反正就是这样了。为什么强调stream呢，stream和http是同级的，http本身有proxy功能，现在stream的核心功能也是proxy。但预料以后stream的功能将更加丰富强大，所以重申nginx有个stream的东西，因为它才具备tcp代理功能。

https://www.zybuluo.com/orangleliu/note/478334 这里也有一些相关的说明

我们的配置中涉及到 proxy_connect_timeout 和 proxy_timeout 两个超时相关的配置项， https://my.oschina.net/xsh1208/blog/199674 这里提到 proxy_connect_timeout 不能超过 75 秒，不过在 https://nginx.org/en/docs/stream/ngx_stream_core_module.html#proxy_protocol_timeout 和 https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_connect_timeout 这两个官方的文档中倒是都没有提到

先写一个 python 的 RabbitMQ 客户端，尝试去读写数据，都能正确连上，但是测试发现一旦闲置超过 1 秒，连接就被断开了，如下：

[x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
 [x] Received 'Hello World!'
Traceback (most recent call last):
  File "mq-receive.py", line 19, in <module>
    channel.start_consuming()
  File "/usr/lib/python2.6/site-packages/pika-0.10.0-py2.6.egg/pika/adapters/blocking_connection.py", line 1681, in start_consuming
    self.connection.process_data_events(time_limit=None)
  File "/usr/lib/python2.6/site-packages/pika-0.10.0-py2.6.egg/pika/adapters/blocking_connection.py", line 647, in process_data_events
    self._flush_output(common_terminator)
  File "/usr/lib/python2.6/site-packages/pika-0.10.0-py2.6.egg/pika/adapters/blocking_connection.py", line 426, in _flush_output
    raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed

于是考虑调整 nginx 的超时配置，简单粗暴的方法当然是配置为从不超时，但是这里有一个问题，就是我们的接入层的超时不应该这么宽松，官方给这些超时配置项的默认值也是到分钟级别的，不过，如果按照这个默认的超时配置值，随之而来的一个问题就是，RabbitMQ 本身是否有心跳呢？如果 RabbitMQ 本身没有心跳，那么在队列不活跃的时候，很可能就被杀掉了

从 https://www.rabbitmq.com/heartbeats.html 这里可以看到

Detecting Dead TCP Connections with Heartbeats

Introduction

Network can fail in many ways, sometimes pretty subtle (e.g. high ratio packet loss). Disrupted TCP connections take a moderately long time (about 11 minutes with default configuration on Linux, for example) to be detected by the operating system. AMQP 0-9-1 offers a heartbeat feature to ensure that the application layer promptly finds out about disrupted connections (and also completely unresponsive peers). Heartbeats also defend against certain network equipment which may terminate “idle” TCP connections.

Heartbeat Timeout Interval

The heartbeat timeout value defines after what period of time the peer TCP connection should be considered dead by RabbitMQ and client libraries. This value is negotiated between the client and RabbitMQ server at the time of connection. The client must be configured to request heartbeats. In RabbitMQ versions 3.0 and higher, the broker will attempt to negotiate heartbeats by default (although the client can still veto them). The timeout is in seconds, and default value is 60 (580 prior to release 3.5.5).

Heartbeat frames are sent about every timeout / 2 seconds. After two missed heartbeats, the peer is considered to be unreachable. Different clients manifest this differently but the TCP connection will be closed. When a client detects that RabbitMQ node is unreachable due to a heartbeat, it needs to re-connect.

Heartbeats can be disabled by setting the timeout interval to 0.

官方是有给 RabbitMQ 3.0 以上的版本支持心跳的，我们用的也是 3.4.1，可以认为是支持的

所以接下来就可以把 nginx 的超时配置改为如下：

stream {
    upstream rabbitmq{
        server xxxx:5672;
    }
    server {
        listen 5672;
        proxy_connect_timeout 600s;
        #proxy_timeout 3s;
        proxy_pass rabbitmq;
    }   
}

然后再次使用模拟的客户端和接收端进行收发，应该就可以保持长期不断开了

UDP 包的边界

Mon, 22 Apr 2013 21:04:40 +0800

首先明确一个问题，如果定义了一个数据结构，大小是，比方说 32 个字节，然后 UDP 客户端连续向服务端发了两个包。现在假设这两个包都已经到达了服务器，那么服务端调用 recvfrom 来接收数据，并且缓冲区开得远大于 64，例如，开了 1024 个字节，那么，服务端的 recvfrom 函数是会一次收到两个数据包呢，还是只能收到一个。

答案是只能收到一个。

来看代码：

struct.h

#ifndef STRUCT_H
#define STRUCT_H

typedef struct _UDP_MSG {
    int add1;
    int add2;
    int sum;
    char str1[16];
    char str2[16];
    char cat[32];
} UDP_MSG;

#endif /* STRUCT_H */

服务器的代码：

#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <unistd.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>

#include "struct.h"

#define MAX_LINE 1024
#define SERV_PORT 8080

int udp_serv();

int main() {
    return udp_serv();
}

int udp_serv() {
    int sockfd = socket(AF_INET, SOCK_DGRAM, 0);
    if (sockfd == -1) {
        perror("socket");
        return -1;
    }
    
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    serv_addr.sin_port = htons(SERV_PORT);

    if (bind(sockfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
        perror("bind");
        return -1;
    }

    while (1) {
        printf("sleeping\n");
        sleep(10);
        printf("akwaked\n");
        char buf[BUFSIZ];
        struct sockaddr_in cli_addr;
        memset(&cli_addr, 0, sizeof(cli_addr));
        socklen_t cli_addr_len = sizeof(cli_addr);
        int recvn = recvfrom(sockfd, buf, sizeof(buf), 0, (struct sockaddr*)&cli_addr, &cli_addr_len);
        printf("recv %d bytes\n", recvn);
        UDP_MSG msg;
        memset(&msg, 0, sizeof(msg));
        printf("UDP_MSG size is %d\n", sizeof(msg));
        memcpy(&msg, buf, sizeof(msg));
        msg.sum = msg.add1 + msg.add2;
        strcpy(msg.cat, msg.str1);
        strcat(msg.cat, msg.str2);
        printf("msg.add1 is: %d\n", msg.add1);
        printf("msg.add2 is: %d\n", msg.add2);
        printf("msg.sum is: %d\n", msg.sum);
        printf("msg.str1 is: %s\n", msg.str1);
        printf("msg.str2 is: %s\n", msg.str2);
        printf("msg.cat is: %s\n", msg.cat);
        sendto(sockfd, &msg, sizeof(msg), 0, (struct sockaddr*)&cli_addr, cli_addr_len);
    }

    return 0;
}

客户端的：

#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <unistd.h>

#include "struct.h"

#define MAX_LINE 1024
#define SERV_PORT 8080

int udp_cli(const char* serv_ip);

int main(int argc, char* argv[]) {
    if (argc < 2) {
        printf("Usage: %s serv_ip\n", argv[0]);
        return 1;
    }
    return udp_cli(argv[1]);
}

int udp_cli(const char* serv_ip) {
    int sockfd = socket(AF_INET, SOCK_DGRAM, 0);
    if (sockfd == -1) {
        perror("socket");
        return -1;
    }
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(SERV_PORT);
    if (inet_pton(AF_INET, serv_ip, &serv_addr.sin_addr) <= 0) {
        perror("inet_pton");
        return -1;
    }
    while (1) {
        UDP_MSG msg;
        memset(&msg, 0, sizeof(msg));
        msg.add1 = 1;
        msg.add2 = 2;
        scanf("%s%s", msg.str1, msg.str2);
        if (sendto(sockfd, &msg, sizeof(msg), 0,
            (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
            perror("write");
            return -1;
        } else {
            printf("send success\n");
        }
        if (sendto(sockfd, &msg, sizeof(msg), 0,
            (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
            perror("write2");
            return -1;
        } else {
            printf("send2 success\n");
        }
        socklen_t serv_addr_len = sizeof(serv_addr);
        if (recvfrom(sockfd, &msg, sizeof(msg), 0,
            (struct sockaddr*)&serv_addr, &serv_addr_len) == -1) {
            perror("read");
            return -1;
        }
        printf("msg.add1 is: %d\n", msg.add1);
        printf("msg.add2 is: %d\n", msg.add2);
        printf("msg.sum is: %d\n", msg.sum);
        printf("msg.str1 is: %s\n", msg.str1);
        printf("msg.str2 is: %s\n", msg.str2);
        printf("msg.cat is: %s\n", msg.cat);
        break;
    }
    return 0;
}

运行起来后，服务端的输出如下：

sleeping
akwaked
recv 76 bytes
UDP_MSG size is 76
msg.add1 is: 1
msg.add2 is: 2
msg.sum is: 3
msg.str1 is: 123
msg.str2 is: 456
msg.cat is: 123456
sleeping
akwaked
recv 76 bytes
UDP_MSG size is 76
msg.add1 is: 1
msg.add2 is: 2
msg.sum is: 3
msg.str1 is: 123
msg.str2 is: 456
msg.cat is: 123456
sleeping

客户端如下：

123
456
send success
send2 success
msg.add1 is: 1
msg.add2 is: 2
msg.sum is: 3
msg.str1 is: 123
msg.str2 is: 456
msg.cat is: 123456

这里涉及到一个边界的问题。 TCP 是流式的数据传输，消息没有边界，需要应用层自己去定义消息边界，而 UDP 是数据报传输，所以协议保证了一次只能接收一个数据报。

详细的解释看到这里，http://hi.baidu.com/chongerfeia/item/c38f91e075d742226dabb8a4

2010-08-11 14:56 有关TCP和UDP 粘包消息保护边界在socket网络程序中，TCP和UDP分别是面向连接和非面向连接的。因此TCP的socket编程，收发两端（客户端和服务器端）都要有一一成对的socket，因此，发送端为了将多个发往接收端的包，更有效的发到对方，使用了优化方法（Nagle算法），将多次间隔较小且数据量小的数据，合并成一个大的数据块，然后进行封包。这样，接收端，就难于分辨出来了，必须提供科学的拆包机制。对于UDP，不会使用块的合并优化算法，这样，实际上目前认为，是由于UDP支持的是一对多的模式，所以接收端的skbuff(套接字缓冲区）采用了链式结构来记录每一个到达的UDP包，在每个UDP包中就有了消息头（消息来源地址，端口等信息），这样，对于接收端来说，就容易进行区分处理了

保护消息边界和流那么什么是保护消息边界和流呢?

保护消息边界，就是指传输协议把数据当作一条独立的消息在网上传输,接收端只能接收独立的消息.也就是说存在保护消息边界,接收端一次只能接收发送端发出的一个数据包. 而面向流则是指无保护消息保护边界的,如果发送端连续发送数据, 接收端有可能在一次接收动作中,会接收两个或者更多的数据包.

我们举个例子来说,例如,我们连续发送三个数据包,大小分别是2k, 4k , 8k,这三个数据包,都已经到达了接收端的网络堆栈中,如果使用UDP协议,不管我们使用多大的接收缓冲区去接收数据,我们必须有三次接收动作,才能够把所有的数据包接收完.而使用TCP协议,我们只要把接收的缓冲区大小设置在14k以上,我们就能够一次把所有的数据包接收下来.只需要有一次接收动作.

这就是因为UDP协议的保护消息边界使得每一个消息都是独立的.而流传输,却把数据当作一串数据流,他不认为数据是一个一个的消息.

所以有很多人在使用tcp协议通讯的时候,并不清楚tcp是基于流的传输,当连续发送数据的时候,他们时常会认识tcp会丢包.其实不然, 因为当他们使用的缓冲区足够大时,他们有可能会一次接收到两个甚至更多的数据包,而很多人往往会忽视这一点,只解析检查了第一个数据包,而已经接收的其他数据包却被忽略了.所以大家如果要作这类的网络编程的时候,必须要注意这一点.

结论：根据以上所说，可以这样理解，TCP为了保证可靠传输，尽量减少额外开销（每次发包都要验证），因此采用了流式传输，面向流的传输，相对于面向消息的传输，可以减少发送包的数量。从而减少了额外开销。但是，对于数据传输频繁的程序来讲，使用TCP可能会容易粘包。当然，对接收端的程序来讲，如果机器负荷很重，也会在接收缓冲里粘包。这样，就需要接收端额外拆包，增加了工作量。因此，这个特别适合的是数据要求可靠传输，但是不需要太频繁传输的场合（两次操作间隔100ms，具体是由TCP等待发送间隔决定的，取决于内核中的socket的写法）

而UDP，由于面向的是消息传输，它把所有接收到的消息都挂接到缓冲区的接受队列中，因此，它对于数据的提取分离就更加方便，但是，它没有粘包机制，因此，当发送数据量较小的时候，就会发生数据包有效载荷较小的情况，也会增加多次发送的系统发送开销（系统调用，写硬件等）和接收开销。因此，应该最好设置一个比较合适的数据包的包长，来进行UDP数据的发送。（UDP最大载荷为1472，因此最好能每次传输接近这个数的数据量，这特别适合于视频，音频等大块数据的发送，同时，通过减少握手来保证流媒体的实时性）

历史评论

wanax (2013-09-23 13:40:08):

哈哈特么这是unix网络编程的课后题啊

ZRJ (2013-09-23 14:10:10):

老子不是科班出身的。。

wanax (2013-09-23 15:34:31):

没这也不是教材别紧张

UDP Socket - itlisten (2017-09-13 23:16:57):

[…] UDP包的边界 […]

Linux 多进程监听 socket

Thu, 11 Apr 2013 15:31:17 +0800

首先来看代码

#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/wait.h>

int main() {
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1 ) {
        perror("socket");
        return -1;
    }
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(8080);
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    if (bind(sockfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
        perror("bind");
        return -1;
    }
    if (listen(sockfd, 20) == -1) {
        perror("listen");
        return -1;
    }
    for (int i = 0; i < 2; i++) {
        pid_t pid = fork();
        if (pid < 0) {
            perror("fork");
        } else if (pid == 0) {
            // child process
            while (1) {
                struct sockaddr_in cli_addr;
                memset(&cli_addr, 0, sizeof(cli_addr));
                socklen_t cli_addr_len = 0;
                int connfd = accept(sockfd, (struct sockaddr*)&cli_addr, &cli_addr_len);
                if (connfd == -1) {
                    perror("accept");
                    return -1;
                }
                char *cli_addr_str = inet_ntoa(cli_addr.sin_addr);
                int cli_addr_port = ntohs(cli_addr.sin_port);
                printf("pid: %d, cli addr: %s:%d\n", getpid(), cli_addr_str, cli_addr_port);
                char buff[4096] = {0};
                if (recv(connfd, buff, sizeof(buff), 0) == -1) {
                    perror("recv");
                    return -1;
                }
                char html_resp[] = "HTTP/1.1 200 OK\r\n\r\nhello";
                if (send(connfd, html_resp, strlen(html_resp), 0) == -1) {
                    perror("send");
                    return -1;
                }
                close(connfd);
            }        
        } else if (pid > 0) {
            // parent process
            printf("forked pid: %d\n", pid);
        }
    }
    while (1) {
        pid_t end_pid = wait(NULL);
        if (end_pid == -1) {
            perror("wait");
            return -1;
        }
        printf("child process %d end\n", end_pid);
    }
    close(sockfd);
    return 0;
}

编译一下

g++ -Wall -g main.cpp -o addrreuse.exe

多次模拟请求

wget http://localhost:8080/123

服务端的输出如下：

forked pid: 23278
forked pid: 23279
pid: 23278, cli addr: 0.0.0.0:0
pid: 23279, cli addr: 0.0.0.0:0
pid: 23278, cli addr: 0.0.0.0:0
pid: 23279, cli addr: 0.0.0.0:0

事实证明，是可以多个进程来 accept 一个 sockfd 的，（好吧，高手当我在说废话就行了），然后注意到有一个东西叫惊群效应，http://blog.csdn.net/chenxinl/article/details/6376895

惊群效应：指一个fd的事件被触发后，等候这个fd的所有线程/进程都被唤醒。低版本的linux内核在accept这边有所谓的“惊群效应”，是指一个链接过来，会把堵塞在accept上的所有子进程都唤醒；但是只有一个子进程都读取到链接，其他子进程唤醒后发现没有数据，继续睡觉去了；这样就产生了大量的进程切换开销。但目前的内核版本已经修复了这个问题，一个链接过来，内核只会唤醒一个子进程出来accept，这样就不用担心惊群效应了。但是对于其他类型的fd，惊群效应还是存在的，而且是必须的；比如多个进程wait在一个pipe的读事件上，有数据过来后，多个进程都会被唤醒，因为内核也不知道这些数据就给一个进程读的，还是每个进程都读一点，这个逻辑是应用层控制的；而accept也相当于一个读事件，但内核确认一个进程接受一个链接就可以了。同理，如果多个子进程把同一个listen的fd添加到各自的epoll_wait中，有链接过来会导致所有的子进程唤醒，这也会导致惊群效应；但这个是应用层面需要解决的问题。关于httpserver惊群效应和如何处理：http://bbs3.chinaunix.net/viewthread.php?action=printable&tid=1676724

但是另外看到这里，http://www.iteye.com/topic/561424

在2.6内核下，内核只会wakeup一个进程。就是说，2.6的内核下面，不会出现惊群的现象。

Linux TCP UDP 混合

Mon, 08 Apr 2013 16:07:14 +0800

基本的思路是这样的：在服务器端，有两个机器，一个对外开启 TCP 监听，然后把监听到的请求内容送到后面的另外一台，或者多台轮询机器上，内网之间使用 UDP，然后等待业务逻辑机器处理完成，这个地方可以做成异步的，然后再返回到用户。

即是这样： client <-TCP-> server_front <-UDP-> server_back

下面来看代码，首先是 server_front 的 demo

#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <arpa/inet.h>

#include "struct.h"

int main() {
    int tcpfd = socket(AF_INET, SOCK_STREAM, 0);
    if (tcpfd == -1) {
        perror("tcpfd socket");
        return -1;
    }
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(38080);
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    if (bind(tcpfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
        perror("tcpfd bind");
        return -1;
    }
    if (listen(tcpfd, 10) == -1) {
        perror("tcpfd listen");
        return -1;
    }
    
    // udp
    int udpfd = socket(AF_INET, SOCK_DGRAM, 0);
    if (udpfd == -1) {
        perror("udpfd socket");
        return -1;
    }
    struct sockaddr_in udp_serv_addr;
    memset(&udp_serv_addr, 0, sizeof(udp_serv_addr));
    udp_serv_addr.sin_family = AF_INET;
    udp_serv_addr.sin_port = htons(8080);
    if (inet_pton(AF_INET, "10.1.164.18", &udp_serv_addr.sin_addr) <= 0) {
        perror("udpfd inet pton");
        return -1;
    }
    
    while (1) {
        int tcp_conn_fd = accept(tcpfd, NULL, NULL);
        if (tcp_conn_fd == -1) {
            perror("tcpfd accept");
            return -1;
        }
        char buff[2048];
        memset(buff, 0, sizeof(buff));
        int recvn = recv(tcp_conn_fd, buff, sizeof(buff), 0);
        if (recvn == -1) {
            perror("tcpfd recv");
            return -1;
        }
        printf("recv: [%s]\n", buff);
        //char html_resp[] = "HTTP/1.1 200 OK\r\n\r\ntcp udp mix";
        UDP_MSG udp_msg;
        memset(&udp_msg, 0, sizeof(udp_msg));
        strcpy(udp_msg.str1, "HTTP/1.1 200 OK\r\n\r\n");
        strcpy(udp_msg.str2, "send to udp for cat");
        if (sendto(udpfd, &udp_msg, sizeof(udp_msg), 0, (struct sockaddr*)&udp_serv_addr, sizeof(udp_serv_addr)) == -1) {
            perror("udp send to");
            return -1;
        }
        if (recvfrom(udpfd, &udp_msg, sizeof(udp_msg), 0, NULL, NULL) == -1) {
            perror("udp recv from");
            return -1;
        }
        if (send(tcp_conn_fd, udp_msg.cat, strlen(udp_msg.cat), 0) == -1) {
            perror("tcp send");
            return -1;
        }
        close(tcp_conn_fd);
    }
    close(tcpfd);
    return 0;
}

然后是内部的报文格式 struct.h

#ifndef STRUCT_H
#define STRUCT_H

typedef struct _UDP_MSG {
    int add1;
    int add2;
    int sum;
    char str1[128];
    char str2[128];
    char cat[256];
} UDP_MSG;

#endif /* STRUCT_H */

然后是后端的业务逻辑服务器

#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>

#include "struct.h"

#define MAX_LINE 1024
#define SERV_PORT 8080

int udp_serv();

int main() {
    return udp_serv();
}

int udp_serv() {
    int sockfd = socket(AF_INET, SOCK_DGRAM, 0);
    if (sockfd == -1) {
        perror("socket");
        return -1;
    }
    
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    serv_addr.sin_port = htons(SERV_PORT);

    if (bind(sockfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) == -1) {
        perror("bind");
        return -1;
    }

    while (1) {
        struct sockaddr_in cli_addr;
        memset(&cli_addr, 0, sizeof(cli_addr));
        UDP_MSG msg;
        memset(&msg, 0, sizeof(msg));
        socklen_t cli_addr_len = sizeof(cli_addr);
        recvfrom(sockfd, &msg, sizeof(msg), 0, (struct sockaddr*)&cli_addr, &cli_addr_len);
        msg.sum = msg.add1 + msg.add2;
        strcpy(msg.cat, msg.str1);
        strcat(msg.cat, msg.str2);
        printf("msg.add1 is: %d\n", msg.add1);
        printf("msg.add2 is: %d\n", msg.add2);
        printf("msg.sum is: %d\n", msg.sum);
        printf("msg.str1 is: %s\n", msg.str1);
        printf("msg.str2 is: %s\n", msg.str2);
        printf("msg.cat is: %s\n", msg.cat);
        sendto(sockfd, &msg, sizeof(msg), 0, (struct sockaddr*)&cli_addr, cli_addr_len);
    }

    return 0;
}

编译一下，就可以测试了

$ wget http://localhost:38080/123 && nl 123
--16:05:03--  http://localhost:38080/123
           => `123.2'
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:38080... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified

    [<=> ] 19            --.--K/s             

16:05:03 (3.62 MB/s) - `123.2' saved [19]

     1	send to udp for cat

期间，server_front 的输出应该是这样的

recv: [GET /123 HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: localhost:38080
Connection: Keep-Alive

]

后台业务逻辑服务器的输出应该是这样的

msg.add1 is: 0
msg.add2 is: 0
msg.sum is: 0
msg.str1 is: HTTP/1.1 200 OK


msg.str2 is: send to udp for cat
msg.cat is: HTTP/1.1 200 OK

send to udp for cat

不同服务器端模型在并发请求下的表现差异

Mon, 25 Mar 2013 11:22:04 +0800

一般来说，服务器的模型有下面几种：

单进程单线程
多进程（一个请求对应一个进程，Apache 之类貌似就是这个）
多线程（一个请求对应一个线程，这个挺少见到的）
select / poll
epoll

它们的差别可以看这个，http://www.cnblogs.com/sharra/archive/2010/12/30/1921287.html

因为需要了解底层设备访问的原理，所以惯用高层应用语言的我，需要了解一下Linux的设备访问机制，尤其是处理一组非阻塞IO的原理方法，标准的术语好像是叫多路复用。以下文章部分句子有引用之处，恕没有一一指出出处。对于接触过Linux内核或设备驱动开发的读者，一定清楚poll和select系统调用，以及从2.5版本引入的epoll机制（epoll机制包含三个系统调用）。网上关于它们的文章，有说用法的，甚为详细，更有分析源代码的，又比较深入，且枝节颇多。经过几篇文章的阅读，我把觉得比较核心的东西写下来吧。我的用意是尽可能以简单的概念，比对他们三者的异同。几经查找我才确定下来，poll和select应该被归类为这样的系统调用，它们可以阻塞地同时探测一组支持非阻塞的IO设备，是否有事件发生（如可读，可写，有高优先级的错误输出，出现错误等等），直至某一个设备触发了事件或者超过了指定的等待时间——也就是它们的职责不是做IO，而是帮助调用者寻找当前就绪的设备。同类型的产品是Windows的IOCP，它也是处理多路复用，只是把IO和探测封装在了一起了。准备的知识有两点：1、fd；2、op->poll。在Linux里面，设备都被抽象为文件，一系列的设备文件就有自己独立的虚拟文件系统，所以，设备在系统调用参数中的表示就是file description。fd其实就是一个整数（特别地，标准输入，输出，错误输出分别对应的fd是0，1，2）。与内核打交道的时候，传递整数的fd可以在自己的文件系统中作进一步的检查是否合法，如果只是返回指针就不能这样操作了，毕竟指针是无差别无意义的。通过fd访问file，通过file可以访问其fileOperator，这里面我们要关心的一个fileOp就是poll。因为系统调用poll和select，就是依靠这个文件操作poll实现的。poll文件操作有两个参数，一个是文件本身，一个可以看做是当设备尚未就绪时调用的回调函数，这个函数是把设备自己特有的等待队列传给内核，让内核把当前的进程挂载到其中（因为当设备就绪时，设备就应该去唤醒在自己特有等待队列中的所有节点，这样当前进程就获取了完成的信号了）。poll文件操作返回的必须是一组标准的掩码，其中的各个位指示当前的不同的就绪状态（全0为没有任何事件触发）。再谈谈早期多路复用的版本poll和select。本质而言，poll和select的共同点就是，对全部指定设备做一次poll，当然这往往都是还没有就绪的，那就会通过回调函数把当前进程注册到设备的等待队列，如果所有设备返回的掩码都没有显示任何的事件触发，就去掉回调函数的函数指针，进入有限时的睡眠状态，再恢复和不断做poll，再作有限时的睡眠，直到其中一个设备有事件触发为止。只要有事件触发，系统调用返回，回到用户态，用户就可以对相关的fd作进一步的读或者写操作了。当然，这个时候还不是所有的设备都就绪的喔，那就得不断地poll或者select了，而做一次这样的系统调用都得轮询所有的设备，次数是设备数*（睡眠次数-1），也就是时间复杂度是O（n），还得做几次O（n）呢。可见，对于现在普遍的服务器程序，需要同时并发监听数千个连接，并且连接需要重复使用的情况，poll和select就存在这样的性能瓶颈。另外，数千个设备fd在每次调用时，都需要将其从用户空间复制到内核空间，这里的开销不可忽略。 poll和select放在一起，是因为其机制一致，而参数和数据结构就略有不同。select一次性传入三组作用于不同信道的设备fd，分别是输入，输出和错误异常。各组的fd期待各组所特有的，由代码指定的一组事件，如输入信道期待输入就绪，输入挂起和错误等事件。然后，select就挑选调用者关心的fd做poll文件操作，检测返回的掩码，看看是否有fd所属信道感兴趣的事件，比如看看这个属于输出信道的fd有没有输出就绪等一系列的事件发生，一样地，如果有一个fd发生感兴趣事件就返回调用了。select，为了同时处理三组使用不同的事件判断规则的fd，采用了位图的方式表示，一组一个位图，位长度是当中最大的fd值，上限是1024，三组就是3072，而且这还只是传入的位图，还有一样大小的传出的位图。当fd数越来越多时，所需的存储开销比较大。既然，一组fd处理起来比较粗放，那就各个fd自己准备好了。poll()系统调用是System V的多元I/O解决方案。它有三个参数，第一个是pollfd结构的数组指针，也就是指向一组fd及其相关信息的指针，因为这个结构包含的除了fd，还有期待的事件掩码和返回的事件掩码，实质上就是将select的中的fd，传入和传出参数归到一个结构之下，也不再把fd分为三组，也不再硬性规定fd感兴趣的事件，这由调用者自己设定。这样，不使用位图来组织数据，也就不需要位图的全部遍历了。按照一般队列地遍历，每个fd做poll文件操作，检查返回的掩码是否有期待的事件，以及做是否有挂起和错误的必要性检查，如果有事件触发，就可以返回调用了。回到poll和select的共同点，面对高并发多连接的应用情境，它们显现出原来没有考虑到的不足，虽然poll比起select又有所改进了。除了上述的关于每次调用都需要做一次从用户空间到内核空间的拷贝，还有这样的问题，就是当处于这样的应用情境时，poll和select会不得不多次操作，并且每次操作都很有可能需要多次进入睡眠状态，也就是多次全部轮询fd，我们应该怎么处理一些会出现重复而无意义的操作。这些重复而无意义的操作有：1、从用户到内核空间拷贝，既然长期监视这几个fd，甚至连期待的事件也不会改变，那拷贝无疑就是重复而无意义的，我们可以让内核长期保存所有需要监视的fd甚至期待事件，或者可以再需要时对部分期待事件进行修改；2、将当前线程轮流加入到每个fd对应设备的等待队列，这样做无非是哪一个设备就绪时能够通知进程退出调用，聪明的开发者想到，那就找个“代理”的回调函数，代替当前进程加入fd的等待队列好了（这也是我后来才总结出来，Linux的等待队列，实质上是回调函数队列吧，也可以使用宏来将当前进程“加入”等待队列，其实就是将唤醒当前进程的回调函数加入队列）。这样，像poll系统调用一样，做poll文件操作发现尚未就绪时，它就调用传入的一个回调函数，这是epoll指定的回调函数，它不再像以前的poll系统调用指定的回调函数那样，而是就将那个“代理”的回调函数加入设备的等待队列就好了，这个代理的回调函数就自己乖乖地等待设备就绪时将它唤醒，然后它就把这个设备fd放到一个指定的地方，同时唤醒可能在等待的进程，到这个指定的地方取fd就好了。我们把1和2结合起来就可以这样做了，只拷贝一次fd，一旦确定了fd就可以做poll文件操作，如果有事件当然好啦，马上就把fd放到指定的地方，而通常都是没有的，那就给这个fd的等待队列加一个回调函数，有事件就自动把fd放到指定的地方，当前进程不需要再一个个poll和睡眠等待了。 epoll机制就是这样改进的了。诚然，fd少的时候，当前进程一个个地等问题不大，可是现在和尚多了，方丈就不好管了。以前设备事件触发时，只负责唤醒当前进程就好了，而当前进程也只能傻傻地在poll里面等待或者循环，再来一次poll，也不知道这个由设备提供的poll性能如何，能不能检查出当前进程已经在等待了就立即返回，当然，我也不明白为什么做了一遍的poll之后，去掉回调函数指针了，还得再做，不是说好了会去唤醒进程的吗？现在就让事件触发回调函数多做一步。本来设备还没就绪就调用一个回调函数了，现在再在这个回调函数里面做一个注册另一个回调函数的操作，目的就是使得设备事件触发多走一步，不仅仅是唤醒当前进程，还要把自己的fd放到指定的地方。就像收本子的班长，以前得一个个学生地去问有没有本子，如果没有，它还得等待一段时间而后又继续问，现在好了，只走一次，如果没有本子，班长就告诉大家去那里交本子，当班长想起要取本子，就去那里看看或者等待一定时间后离开，有本子到了就叫醒他，然后取走。这个道理很简单，就是老师和班干们常说的，大家多做一点工作，我的工作就轻松很多了，尤其是需要管理的东西越来越多时。这种机制或者说模式，我想在Java的FutureTask里面应该也会用到的，一堆在线程池里面跑着的线程（当然这是任务，不是线程，接口是Callable，不是Runnable.run，是Callable.call，它是可以返回结果的），谁先做好就应该先处理呀，可是难道得一个个问吗？干脆就谁好了，谁就按照既定的操作暴露自己，这样FutureTask的get方法就可以马上知道当前最先完成的线程了，就可以取此线程返回结果了。 epoll由三个系统调用组成，分别是epoll_create，epoll_ctl和epoll_wait。epoll_create用于创建和初始化一些内部使用的数据结构；epoll_ctl用于添加，删除或者修改指定的fd及其期待的事件，epoll_wait就是用于等待任何先前指定的fd事件。关于epoll内部的数据结构，我就不能详细了解了。

下面就是分别实现一个 demo，然后用 ab 去压力测试，每回都是 10W 个请求，根据不同的并发量测试

单进程单线程

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;
    char buff[BUFFER_SIZE+1];
    int n;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        memset(&buff, 0, sizeof(buff));
        recv(connfd, buff, BUFFER_SIZE, 0);
        //printf("recv msg:\n%s\n", buff);
        char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
        send(connfd, resp, strlen(resp), 0);
        close(connfd);
    }
    
    close(listenfd);
    return 0;
}

一开始觉得这种简单粗暴的方式肯定不行，结果发现还别说，这东西的处理能力还是可以的，看测试结果

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.210 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13869.50 [#/sec] (mean)
Time per request:       0.072 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)
Transfer rate:          501.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.0      0       4
Waiting:        0    0   0.0      0       4
Total:          0    0   0.0      0       4

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      4 (longest request)
linux:~/ab/httpd-2.4.3/support$

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   5.498 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    18186.97 [#/sec] (mean)
Time per request:       0.550 [ms] (mean)
Time per request:       0.055 [ms] (mean, across all concurrent requests)
Transfer rate:          657.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       6
Processing:     0    0   0.1      0       6
Waiting:        0    0   0.1      0       6
Total:          0    1   0.1      1       6

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      6 (longest request)
linux:~/ab/httpd-2.4.3/support$

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   5.816 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    17194.97 [#/sec] (mean)
Time per request:       5.816 [ms] (mean)
Time per request:       0.058 [ms] (mean, across all concurrent requests)
Transfer rate:          621.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.6      1      11
Processing:     1    5   0.8      5      15
Waiting:        0    2   0.6      2      12
Total:          2    6   0.6      6      15

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      6
  95%      6
  98%      7
  99%      7
 100%     15 (longest request)
linux:~/ab/httpd-2.4.3/support$

150 个并发的时候就不行了

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 150 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 99956 requests completed
linux:~/ab/httpd-2.4.3/support$

接下来是多进程模型

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;
    char buff[BUFFER_SIZE+1];
    int n;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        signal(SIGCHLD, SIG_IGN);
        pid_t pid = fork();
        if (pid < 0) {
            printf("fork error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        } else if (pid == 0) {
            //printf("in child process\n");
            memset(&buff, 0, sizeof(buff));
            recv(connfd, buff, BUFFER_SIZE, 0);
            //printf("recv msg:\n%s\n", buff);
            char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
            send(connfd, resp, strlen(resp), 0);
            close(connfd);
            exit(0);
        } else if (pid > 0) {
            //printf("in parent process\n");
            close(connfd);
            continue;
        }
    }
    
    close(listenfd);
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   41.408 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2415.02 [#/sec] (mean)
Time per request:       0.414 [ms] (mean)
Time per request:       0.414 [ms] (mean, across all concurrent requests)
Transfer rate:          87.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.1      0       9
Waiting:        0    0   0.1      0       9
Total:          0    0   0.1      0       9

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      9 (longest request)
linux:~/ab/httpd-2.4.3/support$

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   39.517 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2530.53 [#/sec] (mean)
Time per request:       3.952 [ms] (mean)
Time per request:       0.395 [ms] (mean, across all concurrent requests)
Transfer rate:          91.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       3
Processing:     0    4   0.4      4      64
Waiting:        0    4   0.4      4      64
Total:          1    4   0.4      4      64

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      4
  99%      4
 100%     64 (longest request)
linux:~/ab/httpd-2.4.3/support$

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   39.980 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2501.28 [#/sec] (mean)
Time per request:       39.980 [ms] (mean)
Time per request:       0.400 [ms] (mean, across all concurrent requests)
Transfer rate:          90.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      17
Processing:     0   40   1.7     40     120
Waiting:        0   40   1.7     40     120
Total:          3   40   1.6     40     120

Percentage of the requests served within a certain time (ms)
  50%     40
  66%     40
  75%     41
  80%     41
  90%     41
  95%     42
  98%     43
  99%     44
 100%    120 (longest request)
linux:~/ab/httpd-2.4.3/support$

500 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 500 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      500
Time taken for tests:   63.222 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    1581.72 [#/sec] (mean)
Time per request:       316.112 [ms] (mean)
Time per request:       0.632 [ms] (mean, across all concurrent requests)
Transfer rate:          57.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   38 520.0      0   21004
Processing:     3  208 2942.9     51   60202
Waiting:        0  208 2942.9     51   60202
Total:         24  247 3153.2     51   63206

Percentage of the requests served within a certain time (ms)
  50%     51
  66%     52
  75%     52
  80%     52
  90%     53
  95%     54
  98%     55
  99%     61
 100%  63206 (longest request)
linux:~/ab/httpd-2.4.3/support$

1000 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1000 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1000
Time taken for tests:   71.625 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    1396.16 [#/sec] (mean)
Time per request:       716.252 [ms] (mean)
Time per request:       0.716 [ms] (mean, across all concurrent requests)
Transfer rate:          50.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   68 628.8      0   21002
Processing:     1  442 4622.4     51   68587
Waiting:        0  442 4622.3     51   68587
Total:         17  510 4934.8     51   71588

Percentage of the requests served within a certain time (ms)
  50%     51
  66%     52
  75%     52
  80%     52
  90%     54
  95%     55
  98%     59
  99%   4107
 100%  71588 (longest request)
linux:~/ab/httpd-2.4.3/support$

再来是多线程

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);
void* process_thread(void* sockfd);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

void* process_thread(void* sockfd) {
    //printf("thread created\n");
    int connfd = *((int *)sockfd);
    char buff[BUFFER_SIZE+1];
    memset(&buff, 0, sizeof(buff));
    recv(connfd, buff, BUFFER_SIZE, 0);
    //printf("recv msg:\n%s\n", buff);
    char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
    send(connfd, resp, strlen(resp), 0);
    close(connfd);
    //printf("thread about to exit\n");
    pthread_exit(NULL);
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        //printf("main thread about to create thread\n");
        pthread_t tid;
        pthread_attr_t opt;
        pthread_attr_init(&opt);
        pthread_attr_setdetachstate(&opt, PTHREAD_CREATE_DETACHED);
        if (pthread_create(&tid, &opt, process_thread, &connfd) != 0) {
            printf("error line: %d\t", __LINE__);
            perror(__FILE__);
            return -1;
        }
        pthread_attr_destroy(&opt);
        //printf("main thread finish creating thread\n");
    }
    
    close(listenfd);
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.629 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13108.37 [#/sec] (mean)
Time per request:       0.076 [ms] (mean)
Time per request:       0.076 [ms] (mean, across all concurrent requests)
Transfer rate:          473.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.1      0      24
Waiting:        0    0   0.1      0      23
Total:          0    0   0.1      0      24

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%     24 (longest request)
linux:~/ab/httpd-2.4.3/support$

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 99991 requests completed
linux:~/ab/httpd-2.4.3/support$

至于 select 和 poll 就没有写了，直接写 epoll 模型

#include <iostream>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>

using namespace std;

#define MAX_LINE 1024
#define OPEN_MAX 100
#define LISTENQ 1024
#define SERV_PORT 8080
#define INFTIM 1000
#define MAX_EVENTS 1024
#define EP_TIMEOUT 500

int tcpSrv(int listen_port);
int set_non_block(int fd);

int main(int argc, char* argv[]) {
    tcpSrv(SERV_PORT);
    return 0;
}

int set_non_block(int fd) {
    int fd_fl = fcntl(fd, F_GETFL); 
    fd_fl |= O_NONBLOCK;
    if (fcntl(fd, F_SETFL, fd_fl) < 0) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        return -1;
    }
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd = socket(AF_INET, SOCK_STREAM, 0);
    if (-1 == listenfd) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        return -1;
    }
    set_non_block(listenfd);

    struct epoll_event ev;
    ev.data.fd = listenfd;
    ev.events = EPOLLIN|EPOLLET;
    int epfd = epoll_create(4096);
    epoll_ctl(epfd, EPOLL_CTL_ADD, listenfd, &ev);

    struct sockaddr_in serveraddr;
    bzero(&serveraddr, sizeof(serveraddr));
    serveraddr.sin_family = AF_INET;
    char* local_addr = "127.0.0.1";
    inet_aton(local_addr, &(serveraddr.sin_addr));
    serveraddr.sin_port = htons(listen_port);

    if (bind(listenfd, (sockaddr*)&serveraddr, sizeof(serveraddr)) == -1) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        close(listenfd);
        return -1;
    }
    
    if (listen(listenfd, LISTENQ) == -1) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        close(listenfd);
        return -1;
    }

    struct epoll_event events[MAX_EVENTS];
    //struct sockaddr_in clientaddr;
    //socklen_t clilen;
    while (1) {
        int nfds = epoll_wait(epfd, events, MAX_EVENTS, EP_TIMEOUT);
        //printf("events count:%d\n", nfds);
        for (int i=0; i<nfds; i++) {
            if (events[i].data.fd == listenfd) {
                while (1) {
                    int connfd = accept(listenfd, NULL, NULL);
                    if (connfd < 0) {
                        //printf("%d\n:", __LINE__);perror(__FILE__);
                        break;
                    }
                    //char *str_client_addr = inet_ntoa(clientaddr.sin_addr);
                    //cout<<"accept a connection from: "<<str_client_addr<<endl;
                    ev.data.fd = connfd;
                    ev.events = EPOLLIN | EPOLLET;
                    epoll_ctl(epfd, EPOLL_CTL_ADD, connfd, &ev);
                }
            } else if (events[i].events & EPOLLIN) {
                //cout<<"EPOLLIN"<<endl;
                int sockfd = events[i].data.fd;
                if (sockfd < 0) {
                    printf("%d\n:", __LINE__);perror(__FILE__);
                    continue;
                }
                char buffer[MAX_LINE+1];
                memset(buffer, 0, sizeof(buffer));
                read(sockfd, buffer, MAX_LINE);
                //printf("%s\n", buffer);
                set_non_block(sockfd);
                char html_resp[] = "HTTP/1.1 200 OK\r\n\r\n<html>epoll</html>";
                write(sockfd, html_resp, strlen(html_resp));
                epoll_ctl(epfd, EPOLL_CTL_DEL, sockfd, NULL);
                close(sockfd);
            }
        }
    }
    
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.474 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13380.33 [#/sec] (mean)
Time per request:       0.075 [ms] (mean)
Time per request:       0.075 [ms] (mean, across all concurrent requests)
Transfer rate:          483.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.0      0       5
Waiting:        0    0   0.0      0       5
Total:          0    0   0.0      0       5

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      5 (longest request)

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   5.991 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    16691.90 [#/sec] (mean)
Time per request:       0.599 [ms] (mean)
Time per request:       0.060 [ms] (mean, across all concurrent requests)
Transfer rate:          603.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       6
Processing:     0    0   0.1      0       7
Waiting:        0    0   0.1      0       6
Total:          0    1   0.2      1       7

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      7 (longest request)

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   6.233 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    16043.99 [#/sec] (mean)
Time per request:       6.233 [ms] (mean)
Time per request:       0.062 [ms] (mean, across all concurrent requests)
Transfer rate:          579.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.0      1       9
Processing:     1    5   0.9      5      12
Waiting:        1    2   0.9      2      10
Total:          2    6   1.4      6      15

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      7
  75%      7
  80%      7
  90%      8
  95%      8
  98%     10
  99%     10
 100%     15 (longest request)

1000 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1000 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1000
Time taken for tests:   7.099 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    14087.17 [#/sec] (mean)
Time per request:       70.987 [ms] (mean)
Time per request:       0.071 [ms] (mean, across all concurrent requests)
Transfer rate:          509.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   26 115.1     19    3054
Processing:    16   44   9.7     43      76
Waiting:        1   18   8.0     18      54
Total:         20   70 115.7     65    3103

Percentage of the requests served within a certain time (ms)
  50%     65
  66%     73
  75%     79
  80%     82
  90%     89
  95%     95
  98%    101
  99%    106
 100%   3103 (longest request)
linux:~/ab/httpd-2.4.3/support$

可以看到 epoll 确实是很猛的，这里有一个地方需要注意，我调了两天，就是在 epoll 的 EPOLLIN 事件发生在 listenfd 上的时候，这个时候需要去 accept，这个应该都没有什么问题，但是我一开始只是 accept 了一个，而实际上，这个时候，是可能有多个连接在等待 accept 的，于是就导致了并发能力很不行，正确的做法应该是一直 accept，直到没有东西可以 accept 了，这样才能有并发能力。

从点击到呈现 — 详解一次HTTP请求（2）

Sun, 30 Sep 2012 16:38:56 +0800

上回说道服务器启动了监听服务，准备迎接来自客户机的请求。那么，我们以一次典型的浏览请求来解析：

当我们开始在浏览器中输入网址的时候，浏览器其实就已经在智能的匹配可能得 url 了，他会从历史记录，书签等地方，找到已经输入的字符串可能对应的 url，然后给出智能提示，对于 google chrome 那种变态的浏览器，他甚至会直接从缓存中把网页展示出来，就是说，你还没有按下 enter，页面就出来了，这个过于奇葩，我们不详细讲。

在例如输入了 baidu 或者 qq 之类的域名后，我们可以按下 ctrl + enter ，来自动补全，生成 qq.com 或者 baidu.com 的网址，然后发起请求。请求一旦发起，浏览器首先要做的事情就是解析这个域名，一般来说，浏览器会首先查看本地硬盘的 hosts 文件，看看其中有没有和这个域名对应的规则，如果有的话就直接使用 hosts 文件里面的 ip 地址，说道这里，大家可能想到，这个地方就存在安全隐患了，如果有病毒把一些常用的域名，修改 hosts 文件，指向一些恶意的 ip，那么浏览器也会不加判断的去连接，是的，这正是很多病毒的惯用手法。

如果在本地的 hosts 文件没有能够找到对应的 ip 地址，浏览器会向 dns 域名解析服务器发起域名解析请求，dns 的域名解析是递归的，（还有另外 dns 是迭代的），递归的 dns 首先会查看自己的 dns 缓存，如果缓存能够命中，那么就从缓存中把 ip 地址返回给浏览器，如果找不到对应的域名的 ip 地址，那么就向上转发请求，然后把得到的这个域名对应的 nameserver 的地址取得，再向这个 namserver 去请求域名对应的 ip，最后把这个 ip 地址返回给浏览器，不过怎么说，这个递归查询的过程，对于浏览器来说是透明的，他只要坐等 ip地址送回来就可以了。对于 DNS 的进一步理解，可以看这里，进一步理解 DNS

得到 ip 地址后，浏览器会开始构造一个 http 请求，一个典型的 http request header 一般需要包括请求的方法，例如 GET 或者 POST 等，不常用的还有 PUT 和 DELETE 方法，更加不常用的还有 HEAD 和 OPTION 以及 TRACE 方法，一般的浏览器只能发起 GET 或者 POST 请求，应用层的 http 请求准备好后，浏览器在传输层发起一条到达服务器的 tcp 连接，这个时候应该开始三次握手的过程，tcp 包被封装到网络层的 ip 包里面，ip 包再被封装到数据链路层的数据帧结构中，再通过物理层的比特流送出去，这些分层的意义在于分工合作，数据链路层通过 CSMA/CD 协议保证了相邻两台主机之间的数据报文传递，而网络层的 ip 数据包通过不同子网之间的路由器的路由算法和路由转发，保证了互联网上两台遥远主机之间的点对点的通讯，不过这种传输是不可靠，于是可靠性就由传输层的 tcp 协议来保证，tcp 通过慢开始，乘法减小等手段来进行流量控制和拥塞避免，同时提供了两台遥远主机上进程到进程的通信，最终保证了 http 的请求头能够被远方的服务器上正在监听的 http 服务器进程收到，终于，数据包在跳与跳之间被拆了又封装，在子网与子网之间被转发了又转发，最后进入了服务器的操作系统的缓冲区，服务器的操作系统由此给正在被阻塞住的 accept 函数一个返回，将他唤醒。

历史评论

smile (2013-09-09 16:57:20):

不是CSMA/CS协议，应该是CSMA/CD吧。

ZRJ (2013-09-09 18:37:07):

呵呵，是的，笔误，修改过来了，谢谢指出哈。。

Jacky (2014-05-02 11:20:15):

DNS部分，好像少了一个，浏览器自身的DNS缓存～

ZRJ (2014-05-03 09:38:38):

嗯，dns 这块确实没讲好，不够清晰易懂，不要被我误导了，哈哈，像 chrome 这种的，还有 chrome://dns/，里面还有 prefetch

老生常谈-从输入url到页面展示到底发生了什么丨 web技术笔记 (2017-09-01 14:38:22):

[…] http://zrj.me/archives/589 […]

从输入URL到页面显示发生了什么 | 言曌博客 (2018-03-07 14:07:34):

[…] http://zrj.me/archives/589 […]

xz (2019-01-28 15:35:08):

有线网络以太网采用的是CSMA/CD，but，无线网络Wi-Fi采用的是CSMA/CA，还是不太一样的。

ZRJ (2019-04-16 10:20:52):

嗯，是的，当时还没学习到无线，现在都是无线为主了，知识老旧了