kk Blog —— 通用基础

curl wget 不验证证书进行https请求

wget ‘https://x.x.x.x/get_ips’ –no-check-certificate

curl ‘https://x.x.x.x/get_ips’ -k

https://blog.csdn.net/hqzxsc2006/article/details/50547684

通过curl得到http各阶段的响应时间

url_effective The URL that was fetched last. This is most meaningful if you’ve told curl to follow location: headers.

filename_effective The ultimate filename that curl writes out to. This is only meaningful if curl is told to write to a file with the –remote-name or –output option. It’s most useful in combination with the –remote-header-name option. (Added in 7.25.1)

http_code http状态码，如200成功,301转向,404未找到,500服务器错误等。(The numerical response code that was found in the last retrieved HTTP(S) or FTP(s) transfer. In 7.18.2 the alias response_code was added to show the same info.)

http_connect The numerical code that was found in the last response (from a proxy) to a curl CONNECT request. (Added in 7.12.4)

time_total 总时间，按秒计。精确到小数点后三位。 （The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.）

time_namelookup DNS解析时间,从请求开始到DNS解析完毕所用时间。(The time, in seconds, it took from the start until the name resolving was completed.)

time_connect 连接时间,从开始到建立TCP连接完成所用时间,包括前边DNS解析时间，如果需要单纯的得到连接时间，用这个time_connect时间减去前边time_namelookup时间。以下同理，不再赘述。(The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.)

time_appconnect 连接建立完成时间，如SSL/SSH等建立连接或者完成三次握手时间。(The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0))

time_pretransfer 从开始到准备传输的时间。(The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.)

time_redirect 重定向时间，包括到最后一次传输前的几次重定向的DNS解析，连接，预传输，传输时间。(The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3))

time_starttransfer 开始传输时间。在client发出请求之后，Web 服务器返回数据的第一个字节所用的时间(The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.)

size_download 下载大小。(The total amount of bytes that were downloaded.)

size_upload 上传大小。(The total amount of bytes that were uploaded.)

size_header 下载的header的大小(The total amount of bytes of the downloaded headers.)

size_request 请求的大小。(The total amount of bytes that were sent in the HTTP request.)

speed_download 下载速度，单位-字节每秒。(The average download speed that curl measured for the complete download. Bytes per second.)

speed_upload 上传速度,单位-字节每秒。(The average upload speed that curl measured for the complete upload. Bytes per second.)

content_type 就是content-Type，不用多说了，这是一个访问我博客首页返回的结果示例(text/html; charset=UTF-8)；(The Content-Type of the requested document, if there was any.)

num_connects Number of new connects made in the recent transfer. (Added in 7.12.3)

num_redirects Number of redirects that were followed in the request. (Added in 7.12.3)

redirect_url When a HTTP request was made without -L to follow redirects, this variable will show the actual URL a redirect would take you to. (Added in 7.18.2)

ftp_entry_path The initial path libcurl ended up in when logging on to the remote FTP server. (Added in 7.15.4)

ssl_verify_result ssl认证结果，返回0表示认证成功。( The result of the SSL peer certificate verification that was requested. 0 means the verification was successful. (Added in 7.19.0))

1、可以直接访问使用：

1

curl -o /dev/null -s -w %{http_code}:%{http_connect}:%{content_type}:%{time_namelookup}:%{time_redirect}:%{time_pretransfer}:%{time_connect}:%{time_starttransfer}:%{time_total}:%{speed_download} www.baidu.com

输出变量需要按照%{variable_name}的格式，如果需要输出%，double一下即可，即%%，同时，\n是换行，\r是回车，\t是TAB。

-w 指定格式化文件

-o 请求重定向到,不带此参数则控制台输出返回结果

-s 静默，不显示进度

2、也可以定义时间格式化文件访问

1
2
3
4
5
6
7
8
9
10
11
12
13
14

#vim  curl-time.txt 
\n
            http: %{http_code}\n
               dns: %{time_namelookup}s\n
          redirect: %{time_redirect}s\n
      time_connect: %{time_connect}s\n
   time_appconnect: %{time_appconnect}s\n
  time_pretransfer: %{time_pretransfer}s\n
time_starttransfer: %{time_starttransfer}s\n
     size_download: %{size_download}bytes\n
    speed_download: %{speed_download}B/s\n
                  ----------\n
        time_total: %{time_total}s\n
\n

1

curl -w "@curl_time.txt"  -s  -H "Content-Type: application/json" --insecure --header 'Host: passport.500.com' --data '{"platform":"android","userimei":"F5D815EA2BD8DBARD","app_channel":"10000","mbimei":"9DB358AF","version":"3.1.4","username":"hqzx","userpass":"976af4"}' --compressed https://119.147.113.177/user/login

http://297020555.blog.51cto.com/1396304/592386

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

*/1 * * * * cd /root/test ; ./curl.sh > /dev/null 2>&1
*/1 * * * * cd /root/test ; ./ping.sh > /dev/null 2>&1

curl.sh:
if ps aux | grep curl | grep "30M" > /dev/null ; then
	echo "..." > /dev/null
else
	./curl_1.sh &
	./curl_2.sh &
fi

curl_1.sh & curl_2.sh:
URL=192.168.1.3:80/30M
T=`date "+%F %T"`
curl $URL -s -o /tmp/df -w "$T %{time_connect} %{time_starttransfer} %{time_total} %{speed_download} %{http_code}\n" >> t_down

ping.sh:
date "+%F %T" >> ping_out
ping -q -f -c 1000 192.168.1.3 >> ping_out

res.sh:

1
2
3
4
5
6

#sed -i -e ':a;N;$!ba;s/\(:..\)\n/\1 /g' t_down
cat ping_out | grep -E "\-|loss" | grep -v statistics > ping_tmp
sed -i -e ':a;N;$!ba;s/\(:..\)\n/\1 /g' ping_tmp

python res.py > res_out
python res2.py

统计结果和ping汇聚 res.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

import time

ff = open('t1_down').readlines()
qq = open('t2_down').readlines()
ping = open('ping_tmp').readlines()

j = 0
for i in range(0, len(ff)):
	f = ff[i].strip().split(' ')
	q = qq[i].strip().split(' ')

	if (len(f) != len(q)):
		break

	timeArray = time.strptime(f[0]+" "+f[1], "%Y-%m-%d %H:%M:%S")
	ft = time.mktime(timeArray)

	if (i < len(ff) - 1):
		fn = ff[i+1].strip().split(' ')
		fnt = time.mktime(time.strptime(fn[0] + " " + fn[1], "%Y-%m-%d %H:%M:%S"))
	else:
		fnt = time.mktime(time.strptime("2020-01-01 00:00:00", "%Y-%m-%d %H:%M:%S"))

	po = "0"
	while (j < len(ping)):
		p = ping[j].strip().split(' ')
		pt = time.mktime(time.strptime(p[0]+" "+p[1], "%Y-%m-%d %H:%M:%S"))
		if (pt > fnt):
			break
		j = j + 1
		if (pt >= ft - 10):
			po = p[8]
			break

	print f[0], f[1], int(float(f[6])/1000), int(float(q[6])/1000), po

分时段统计res2.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#coding:utf-8

r = open('res_out').readlines()

hs1 = {}
hs2 = {}
hsw = {}
hl = {}
hr = {}
hm = {}

for line in r:
	arr = line.strip().split(' ')
	h1 = int(arr[1][0:2])
	if h1 >= 2 and h1 <= 9:
		h2 = "H2-9"
	else:
		h2 = "H10-25"
	h3 = 'Hall'
	for h in (h1, h2, h3):
		if h not in hsw:
			hs1[h] = hs2[h] = hsw[h] = hl[h] = hr[h] = hm[h] = 0
		hs1[h] += int(arr[2])
		hs2[h] += int(arr[3])
		hsw[h] += 1
		hl[h] += int(arr[4][0:-1])
		hr[h] += int(arr[5])
		hm[h] += int(arr[6])

print "时段", "下载次数", "ff(KB/s)", "qq(KB/s)", "提升比例", "丢包率", "recovery_skb", "mark_skb"
for h in sorted(hs1.keys()):
	s1 = hs1[h]/hsw[h]
	s2 = hs2[h]/hsw[h]
	s3 = hl[h]/hsw[h]
	s4 = hr[h]/hsw[h]
	s5 = hm[h]/hsw[h]
	#print h, hsw[h], s1, s2, 100*(s1-s2)/s2, s3, s4, s5
	print "%s\t%d\t%d\t%d\t%d\t%d\t%d\t%d" % (h, hsw[h], s1, s2, 100*(s1-s2)/s2, s3, s4, s5)

一、http_load

http_load以并行复用的方式运行，用以测试web服务器的吞吐量与负载。但是它不同于大多数压力测试工具，它可以以一个单一的进程运行，一般不会把客户机搞死。还可以测试HTTPS类的网站请求。

下载地址：http://www.acme.com/software/http_load/

1

./http_load -verbose -proxy 192.168.99.6:80 -parallel 24 -seconds 1000 url.txt

http_load 改进版下载 http_load-09Mar2016-kk.tar.gz
改进点：
2018-01-19:
1. 异常时打印更多信息(“want_bytes=%ld got_bytes=%ld sport=%d connect_at=%ld now=%ld last=%ld”)
2. http1.0 改成 http1.1 支持多次request
3. 增加 [ -requests times ] 参数, 在一条流中会发起times次request, 默认为1
2018-01-26:
4. 增加 [ -fastopen ] 参数，http协议增加fastopen测试，fastopen连接时改为阻塞模式。非阻塞模式syn无法附带数据
2018-04-12:
5. 修复 num_connections 可能出现的统计错误，以及fastopen可能出现的请求超时
2018-06-13:
6. https增加SNI信息，Makefile默认开启https支持
2019-02-19:
7. 修复IPV6的bug
2020-08-29: 8. fastopen 支持非阻塞模式-nonblock，但第一次连接还得用block

二、webbench

webbench是Linux下的一个网站压力测试工具，最多可以模拟3万个并发连接去测试网站的负载能力。

1
2

用法：webbench -c 并发数 -t 运行测试时间 URL
如：webbench -c 5000 -t 120 http://www.163.com

三、ab

ab是apache自带的一款功能强大的测试工具。安装了apache一般就自带了，用法可以查看它的说明

参数众多，一般我们用到的是-n 和-c

例如：

1

./ab -c 1000 -n 100 http://www.vpser.net/index.php

这个表示同时处理1000个请求并运行100次index.php文件.

四、Siege

一款开源的压力测试工具，可以根据配置对一个WEB站点进行多用户的并发访问，记录每个用户所有请求过程的相应时间，并在一定数量的并发访问下重复进行。 官方：http://www.joedog.org/

使用

1

siege -c 200 -r 10 -f example.url

-c是并发量，-r是重复次数。 url文件就是一个文本，每行都是一个url，它会从里面随机访问的。

两个数A B，A<B，两种操作：A=A+1 或 A=A*2，求A到B的最少操作次数。

首先如果A、B的二进制前缀不一样则一直A=A+1

然后A=A<<1，A、B前缀不一样再A=A+1

blog.csdn.net/zhangskd/article/details/7609465

概述

问题1：当发送方长时间受到应用程序的限制，不能发送数据时，会使拥塞窗口无效。TCP是根据拥塞窗口来动态地估计网络带宽的。发送方受到应用程序的限制后，没有数据可以发送。那么此时的拥塞窗口就不能准确的反应网络状况，因为这个拥塞窗口是很早之前的。

问题2：当发送方受到应用程序限制，不能利用完拥塞窗口，会使拥塞窗口的增长无效。TCP不断调整cwnd来测试网络带宽。如果不能完全使用掉cwnd，就不知道网络能否承受得了cwnd的数据量，这种情况下的cwnd增长是无效的。

原理

TCP sender受到的两种限制

(1) application-limited ：when the sender sends less than is allowed by the congestion or receiver window.

(2) network-limited：when the sender is limited by the TCP window. More precisely, we define a network-limited period as any period when the sender is sending a full window of data.

问题1描述

TCP’s congestion window controls the number of packets a TCP flow may have in the network at any time. However, long periods when the sender is idle or application-limited can lead to the invalidation of the congestion window, in that the congestion window no longer reflects current information about the state of the network.

The congestion window is set using an Additive-Increase, Multiplicative-Decrease(AIMD) mechanism that probes for available bandwidth, dynamically adapting to changing network conditions. This AIMD works well when the sender continually has data to send, as is typically the case for TCP used for bulk-data transfer. In contrast, for TCP used with telnet applications, the data sender often has little or no data to send, and the sending rate is often determined by the rate at which data is generated by the user.

问题2描述

An invalid congestion window also results when the congestion window is increased (i.e., in TCP’s slow-start or congestion avoidance phases) during application-limited periods, when the previous value of the congestion window might never have been fully utilized. As far as we know, all current TCP implementations increase the congestion window when an acknowledgement arrives, if allowed by the receiver’s advertised window and the slow-start or congestion avoidance window increase algorithm, without checking to see if the previous value of the congestion window has in fact been used.

This document proposes that the window increase algorithm not be invoked during application- limited periods. This restriction prevents the congestion window from growing arbitrarily large, in the absence of evidence that the congestion window can be supported by the network.

实现(1)

发送方在发送数据包时，如果发送的数据包有负载，则会检测拥塞窗口是否超时。如果超时，则会使拥塞窗口失效并重新计算拥塞窗口。然后根据最近接收段的时间，确定是否进入pingpong模式。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

/* Congestion state accounting after a packet has been sent. */  
static void tcp_event_data_sent (struct tcp_sock *tp, struct sock *sk)  
{  
	struct inet_connection_sock *icsk = inet_csk(sk);  
	const u32 now = tcp_time_stamp;  
  
	if (sysctl_tcp_slow_start_after_idle &&   
		(!tp->packets_out && (s32) (now - tp->lsndtime) > icsk->icsk_rto))  
		tcp_cwnd_restart(sk, __sk_dst_get(sk)); /* 重置cnwd */  
  
	tp->lsndtime = now; /* 更新最近发包的时间*/  
  
	/* If it is a reply for ato after last received packets,  
	 * enter pingpong mode. */  
	if ((u32)(now - icsk->icsk_ack.lrcvtime) < icsk.icsk_ack.ato)  
		icsk->icsk_ack.pingpong = 1;  
}  

tcp_event_data_sent()中，符合三个条件才重置cwnd：

（1）tcp_slow_start_after_idle选项设置，这个内核默认置为1 （2）tp->packets_out == 0，表示网络中没有未确认数据包 （3）now - tp->lsndtime > icsk->icsk_rto，距离上次发送数据包的时间超过了RTO

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

/* RFC2861. Reset CWND after idle period longer RTO to "restart window". 
 * This is the first part of cnwd validation mechanism. 
 */  
static void tcp_cwnd_restart (struct sock *sk, const struct dst_entry *dst)  
{  
	struct tcp_sock *tp = tcp_sk(sk);  
	s32 delta = tcp_time_stamp - tp->lsndtime;  
  
	/* 关于tcp_init_cwnd()可见上一篇blog.*/  
	u32 restart_cwnd = tcp_init_cwnd(tp, dst);  
	u32 cwnd = tp->snd_cwnd;  
	  
	/* 触发拥塞窗口重置事件*/  
	tcp_ca_event(sk, CA_EVENT_CWND_RESTART);  
  
	/* 阈值保存下来，并没有重置。*/  
	tp->snd_ssthresh = tcp_current_ssthresh(sk);  
	restart_cwnd = min(restart_cwnd, cwnd);  
  
	/* 闲置时间每超过一个RTO且cwnd比重置后的大时，cwnd减半。*/  
	while((delta -= inet_csk(sk)->icsk_rto) > 0 && cwnd > restart_cwnd)  
		cwnd >> 1;  
  
	tp->snd_cwnd = max(cwnd, restart_cwnd); /* 取其大者！*/  
	tp->snd_cwnd_stamp = tcp_time_stamp;  
	tp->snd_cwnd_used = 0;  
}  

那么调用tcp_cwnd_restart()后，tp->snd_cwnd是多少呢？这个是不确定的，要看闲置时间delta、闲置前的cwnd、路由器中设置的initcwnd。当然，最大概率的是：拥塞窗口降为闲置前cwnd的一半。

实现(2)

在发送方成功发送一个数据包后，会检查从发送队列发出而未确认的数据包是否用完拥塞窗口。 如果拥塞窗口被用完了，说明发送方收到网络限制； 如果拥塞窗口没被用完，且距离上次检查时间超过了RTO，说明发送方收到应用程序限制。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

/* Congestion window validation.(RFC2861) */  
static void tcp_cwnd_validate(struct sock *sk) {  
	struct tcp_sock *tp = tcp_sk(sk);  
  
	if (tp->packets_out >= tp->snd_cwnd) {  
		/* Network is feed fully. */  
		tp->snd_cwnd_used = 0; /*不用这个变量*/  
		tp->snd_cwnd_stamp = tcp_time_stamp; /* 更新检测时间*/  
  
	} else {  
		/* Network starves. */  
		if (tp->packets_out > tp->snd_cwnd_used)  
			tp->snd_cwnd_used = tp->packets_out; /* 更新已使用窗口*/  
  
			/* 如果距离上次检测的时间，即距离上次发包时间已经超过RTO*/  
			if (sysctl_tcp_slow_start_after_idle &&  
				(s32) (tcp_time_stamp - tp->snd_cwnd_stamp) >= inet_csk(sk)->icsk_rto)  
				tcp_cwnd_application_limited(sk);  
	}  
}  

在发送方收到应用程序的限制期间，每隔RTO时间，都会调用tcp_cwnd_application_limited()来重新设置sshresh和cwnd，具体如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

/* RFC2861, slow part. Adjust cwnd, after it was not full during one rto. 
 * As additional protections, we do not touch cwnd in retransmission phases, 
 * and if application hit its sndbuf limit recently. 
 */  
void tcp_cwnd_application_limited(struct sock *sk)  
{  
	struct tcp_sock *tp = tcp_sk(sk);  
  
	/* 只有处于Open态，应用程序没受到sndbuf限制时，才进行 
	 * ssthresh和cwnd的重置。 
	 */  
	if (inet_csk(sk)->icsk_ca_state == TCP_CA_Open &&   
		sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) {  
  
		/* Limited by application or receiver window. */  
		u32 init_win = tcp_init_cwnd(tp, __sk_dst_get(sk));  
		u32 win_used = max(tp->snd_cwnd_used, init_win);  
  
		/* 没用完拥塞窗口*/  
		if (win_used < tp->snd_cwnd) {  
			/* 并没有减小ssthresh，反而增大，保留了过去的信息，以便之后有数据发送 
			  * 时能快速增大到接近此时的窗口。 
			  */  
			tp->snd_ssthresh = tcp_current_ssthresh(sk);   
			/* 减小了snd_cwnd */  
			tp->snd_cwnd = (tp->snd_cwnd + win_used) >> 1;  
		}  
		tp->snd_cwnd_used = 0;  
	}  
	tp->snd_cwnd_stamp = tcp_time_stamp; /* 更新最近的数据包发送时间*/  
}  

发送方受到应用程序限制，且限制的时间每经过RTO后，就会调用以上函数来处理snd_ssthresh和snd_cwnd：

（1）snd_ssthresh = max(snd_ssthresh, ¾ cwnd)

慢启动阈值并没有减小，相反，如果此时cwnd较大，ssthresh会相应的增大。ssthresh是一个很重要的参数，它保留了旧的信息。这样一来，如果应用程序产生了大量的数据，发送方不再受到限制后，经过慢启动阶段，拥塞窗口就能快速恢复到接近以前的值了。

（2）snd_cwnd = (snd_cwnd + snd_cwnd_used) / 2

因为snd_cwnd_used < snd_cwnd，所以snd_cwnd是减小了的。减小snd_cwnd是为了不让它盲目的增长。因为发送方没有利用完拥塞窗口，并不能检测到网络是否能承受该拥塞窗口，这时的增长是无根据的。

结论

在发送完数据包后，通过对拥塞窗口有效性的检验，能够避免使用不合理的拥塞窗口。

拥塞窗口代表着网络的状况，通过避免使用不合理的拥塞窗口，就能得到正确的网络状况，而不会采取一些不恰当的措施。

在上文的两种情况下，通过TCP的拥塞窗口有效性验证机制（TCP congestion window validationmechanism），能够更合理的利用网络、避免丢包，从而提高传输效率。

Reference

RFC2861