SpringBoot-线上服务卡死卡顿分析-http请求进不去服务-调整tomcat-undertow线程数
线程监控工具查看 stack.log文件
查内存镜像和阻塞情况：
java调优相关：来源：https://gitee.com/jiangjiesheng/java-optimization-and-debug-for-prod-env
jps -l 查看当前系统运行了哪些Java进程【给出进程号和包名】
jstat 查看jvm统计信息（好像没啥实际的作用）
jmap 手动导出内存镜像文件用于内存溢出分析
（结合jps -l 或ps -ef | grep 查到进程号）
jmap -dump:format=b,file=java_pid9544_by_jmap.hprof 9544，
也可以通过参数自动导出 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./export/
分析工具：MemoryAnalyzer.exe（MAT工具）
jstack 实战死循环与死锁 ，由阻塞方法引起的死锁，看起来不动了【实用】
jstack -F 进程号 > jstack.deadlock.10892.txt # -F参数可以强制导出
或者 jstack -l 进程号 | grep deadlocks 进程号，直接展示结果。
报告中搜索 deadlock 关键词，如果没有死锁 No deadlocks found。
如果报告中有 Thread 1783: (state = BLOCKED)，这个怎么定位？结合服务的log日志中线程号来定位？
不处理的危害可能导致java占用内存越来越大，这个时候就需要去脚本限制内存大小了
jstack 8 > stack.log
查SQL情况：
mysql -hmysql -umaster -ppZdxysqJX4bK4VrB -A
edu:
use seats
SELECT trx_mysql_thread_id,trx_query FROM INFORMATION_SCHEMA.INNODB_TRX
where trx_mysql_thread_id in (select id from information_schema.processlist where command != 'Sleep' )
and (trx_query like "%course%" or trx_query like "%edu%" or trx_query like "%exam%" or trx_query like "%inte%")\G
如何排查java程序导致的cpu和内存过高异常
https://blog.csdn.net/qq_40322236/article/details/127223538
打印gc日志：
-Dlogin.cache.switch=true -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./
undertow设置：
-DinvokeSeconds=60 -Dserver.undertow.io-threads=16 -Dserver.undertow.worker-threads=500
或
server:
undertow:
io-threads: 16
worker-threads: 500
实测：无效，断点看相关的初始化还是按默认的计算规则走的
tomcat设置：
-Dserver.tomcat.max-threads=500 -Dserver.tomcat.max-connections=20000
-Dserver.tomcat.accept-count=2000 -Dserver.tomcat.min-spare-threads=20
下次卡的时候 再看线程
另外看undertow的 java.lang.Thread.State: TIMED_WAITING (parking)
处理1
通过代码初始化线程数，debug初始化有效
package cn.jiangjiesheng.edu.config;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.web.embedded.undertow.UndertowServletWebServerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class UndertowConfig {
@Value(value = "${server.undertow.io-threads:16}")
private Integer ioThreads;
@Value(value = "${server.undertow.worker-threads:500}")
private Integer workerThreads;
@Bean
public UndertowServletWebServerFactory embeddedServletContainerFactory() {
UndertowServletWebServerFactory factory = new UndertowServletWebServerFactory();
factory.setIoThreads(ioThreads);
factory.setWorkerThreads(workerThreads);
return factory;
}
}
处理2：
加锁方式，调试不同人员的学习 会 触发 同一个课程id的数据同步，
所以同一课程处理处理过程中要排除掉进来的其他请求【可能也有问题，极低概率下正好错过了】
concurrentTaskBySingleKeyExecutor.doTask("syncEduTrainImplementRecordDetailService
#insertOrUpdateCourse", implementId + "#courseId" + courseId, new Runnable() {
网络资料：
https://www.zhihu.com/tardis/bd/art/401186598?source_id=1001
【关键，但是参数设置，不生效，要用代码初始化，根据断点看Undertow的初始化】
debug关键代码：
private Undertow(Builder builder) {
...
this.ioThreads = builder.ioThreads;
this.workerThreads = builder.workerThreads;
...
}
public synchronized void start() {
UndertowLogger.ROOT_LOGGER.infof("starting server: %s", Version.getFullVersionString());
xnio = Xnio.getInstance(Undertow.class.getClassLoader());
channels = new ArrayList<>();
try {
if (internalWorker) {
worker = xnio.createWorker(OptionMap.builder()
.set(Options.WORKER_IO_THREADS, ioThreads)
.set(Options.CONNECTION_HIGH_WATER, 1000000)
.set(Options.CONNECTION_LOW_WATER, 1000000)
.set(Options.WORKER_TASK_CORE_THREADS, workerThreads)
.set(Options.WORKER_TASK_MAX_THREADS, workerThreads)
.set(Options.TCP_NODELAY, true)
.set(Options.CORK, true)
.addAll(workerOptions)
.getMap());
}
....
}
public static final class Builder {
private int ioThreads;
private int workerThreads;
private Builder() {
ioThreads = Math.max(Runtime.getRuntime().availableProcessors(), 2);
workerThreads = ioThreads * 8;
long maxMemory = Runtime.getRuntime().maxMemory();
//smaller than 64mb of ram we use 512b buffers
if (maxMemory < 64 * 1024 * 1024) {
//use 512b buffers
directBuffers = false;
bufferSize = 512;
} else if (maxMemory < 128 * 1024 * 1024) {
//use 1k buffers
directBuffers = true;
bufferSize = 1024;
} else {
//use 16k buffers for best performance
//as 16k is generally the max amount of data that can be sent in a single write() call
directBuffers = true;
bufferSize = 1024 * 16 - 20; //the 20 is to allow some space for protocol headers, see UNDERTOW-1209
}
}
public Builder setIoThreads(final int ioThreads) {
this.ioThreads = ioThreads;
return this;
}
public Builder setWorkerThreads(final int workerThreads) {
this.workerThreads = workerThreads;
return this;
}
}
更多网络资料：
https://blog.csdn.net/weixin_43958014/article/details/129684017
https://www.lxiaoyu.com/p/161092
https://www.cnblogs.com/koal/p/12347982.html
https://blog.csdn.net/tiandixuanwuliang/article/details/102733441
https://www.ibm.com/support/pages/ibm-thread-and-monitor-dump-analyzer-java-tmda
正文到此结束
- 本文标签： Spring Boot Spring
- 本文链接： https://code.jiangjiesheng.cn/article/32
- 版权声明： 本文由小江同学原创发布，转载请先联系本站长，谢谢。
