None
When Continuously Waiting to Receive Messages from a Remote Server
The thread appears to be normal,since its state keeps showing as RUNNABLE. However,when you align the thread dumps chronologically,you can see that socketReadThread thread is waiting infinitely to read the socket.
figure 5: Continuous Waiting Status.
Highlighter_319824" class="
SyntaxHighlighter ">
"socketReadThread" prio=6 tid=0x0000000006a0d800 nid=0x1b40 runnable [0x00000000089ef000]
java.lang.Thread.State: RUNNABLE
at java.net
.socketInputStream
.socketRead0(Native Method)
at java.net
.socketInputStream.read(SocketInputStream.java:129)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
locked (a java.io.InputStreamReader)
at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
locked (a java.io.InputStreamReader)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
at java.io.InputStreamReader.read(InputStreamReader.java:151)
at com.nbp.theplatform.threaddump.ThreadSocketReadState$1.run(ThreadSocketReadState.java:27)
at java.lang.Thread.run(Thread.java:662)
When Waiting
The thread is maintaining WAIT status. In the thread dump,IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue,then the thread status will not change.
figure 6: Waiting status.
Highlighter_973678" class="
SyntaxHighlighter ">
"IoWaitThread" prio=6 tid=0x0000000007334800 nid=0x2b3c waiting on condition [0x000000000893f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:440)
at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:629)
at com.nbp.theplatform.threaddump.ThreadioWaitState$IoWaitHandler2.run(ThreadioWaitState.java:89)
at java.lang.Thread.run(Thread.java:662)
nes">
When Thread Resources Cannot be Organized normally
Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs,it is recommended to monitor the thread organization process or check the conditions for thread termination.
norganized Threads" src="http://www.cubrid.org/files/attach/images/220547/971/295/when-thread-resources-cannot-be-organized-normally.png" alt=" Unorganized Threads" width="678" height="193">
figure 7: Unorganized Threads.
How to Solve Problems by Using Thread Dump
Example 1: When the cpu Usage is Abnormally High
1. Extract the thread that has the highest cpu usage.
Highlighter_839197" class="
SyntaxHighlighter ">
[user@linux ~]$
-mo pid.lwp.stime..cpu - PID LWP STIME TIME </span>%<span style="color: #000000;"><a href="https://www.jb51.cc/tag/cpu/" target="_blank" class="keywords">cpu</a>
<span style="color: #800080;">10029 - Dec07 <span style="color: #800080;">00:<span style="color: #800080;">02:<span style="color: #800080;">02 <span style="color: #800080;">99.5
From the application,find out which thread is using the cpu the most.
Acquire the Light Weight Process (LWP) that uses the cpu the most and convert its unique number (10039) into a hexadecimal number (0x2737).
2. After acquiring the thread dump,check the thread's action.
Extract the thread dump of an application with a PID of 10029,then find the thread with an nid of 0x2737.
Highlighter_746899" class="
SyntaxHighlighter ">
"NioProcessor-2" prio=10 tid=0x0a8d2800 nid=0x2737 runnable [0x49aa5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked (a sun.nio.ch.Util$1)
- locked (a java.util.Collections$UnmodifiableSet)
- locked (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at external.org.apache.mina.transport
.socket.nio.NioProcessor.select(NioProcessor.java:65)
at external.org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:708)
at external.org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at
java.lang.Thread.run(Thread.java:662)
Extract thread dumps several times every hour,and check the status change of the threads to determine the problem.
Example 2: When the Processing Performance is Abnormally Slow
After acquiring thread dumps several times,find the list of threads with BLOCKED status.
Highlighter_103913" class="
SyntaxHighlighter ">
" DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
- waiting to lock (a beans.ConnectionPool)
at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)
"DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020]
java.lang.Thread.State: BLOCKED (on object monitor)
at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
- waiting to lock (a beans.ConnectionPool)
at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)
" DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570)
- waiting to lock (a oracle.jdbc.driver.OracleConnection)
at beans.ConnectionPool.getConnection(ConnectionPool.java:112)
- locked (a java.util.Vector)
- locked (a beans.ConnectionPool)
at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66)
at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120)
nes">
Acquire the list of threads with BLOCKED status after getting the thread dumps several times.
If the threads are BLOCKED,extract the threads related to the lock that the threads are trying to obtain.
Through the thread dump,you can confirm that the thread status stays BLOCKED because <0xe0375410> lock Could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock.
There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working,they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them,you will often see that some of the threads that were BLOCKED prevIoUsly are in a different state.
The second reason is the abnormal connection. When the connection with DBMS stays abnormal,the threads wait until the time is out. In this case,even after extracting the thread dumps several times and comparing them,you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values,such as the timeout value,you can shorten the time in which the problem occurs.
Coding for Easy Thread Dump
Naming Threads
When a thread is created using java.lang.Thread object,the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object,the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application,if all the threads still have their default names,analyzing them becomes very difficult,because it is difficult to distinguish the threads to be analyzed.
Therefore,you are recommended to develop the habit of naming the threads whenever a new thread is created.
When you create a thread using java.lang.Thread,you can give the thread a custom name by using the creator parameter.
Highlighter_998691" class="
SyntaxHighlighter ">
Thread(ThreadGroup group,String name, stackSize);
When you create a thread using java.util.concurrent.ThreadFactory,you can name it by generating your own ThreadFactory. If you do not need special functionalities,then you can use MyThreadFactory as described below:
Highlighter_50112" class="
SyntaxHighlighter ">
<span style="color: #0000ff;">public <span style="color: #0000ff;">class MyThreadFactory <span style="color: #0000ff;">implements<span style="color: #000000;"> ThreadFactory {
<span style="color: #0000ff;">private <span style="color: #0000ff;">static <span style="color: #0000ff;">final ConcurrentHashMap<String,AtomicInteger> POOL_NUMBER =
<span style="color: #0000ff;">new ConcurrentHashMap<String,AtomicInteger><span style="color: #000000;">();
<span style="color: #0000ff;">private <span style="color: #0000ff;">final<span style="color: #000000;"> ThreadGroup group;
<span style="color: #0000ff;">private <span style="color: #0000ff;">final AtomicInteger threadNumber = <span style="color: #0000ff;">new AtomicInteger(1<span style="color: #000000;">);
<span style="color: #0000ff;">private <span style="color: #0000ff;">final<span style="color: #000000;"> String namePrefix;
<span style="color: #0000ff;">public<span style="color: #000000;"> MyThreadFactory(String threadPoolName) {
</span><span style="color: #0000ff;">if</span> (threadPoolName == <span style="color: #0000ff;">null</span><span style="color: #000000;">) {
</span><span style="color: #0000ff;">throw</span> <span style="color: #0000ff;">new</span> NullPointerException("threadPoolName"<span style="color: #000000;">);
}
POOL_NUMBER.putIfAbsent(threadPoolName,</span><span style="color: #0000ff;">new</span><span style="color: #000000;"> AtomicInteger());
S<a href="https://www.jb51.cc/tag/ecurity/" target="_blank" class="keywords">ecurity</a>Manager s<a href="https://www.jb51.cc/tag/ecurity/" target="_blank" class="keywords">ecurity</a>Manager </span>=<span style="color: #000000;"> Sy<a href="https://www.jb51.cc/tag/stem/" target="_blank" class="keywords">stem</a>.getS<a href="https://www.jb51.cc/tag/ecurity/" target="_blank" class="keywords">ecurity</a>Manager();
group </span>= (s<a href="https://www.jb51.cc/tag/ecurity/" target="_blank" class="keywords">ecurity</a>Manager != <span style="color: #0000ff;">null</span>) ?<span style="color: #000000;"> s<a href="https://www.jb51.cc/tag/ecurity/" target="_blank" class="keywords">ecurity</a>Manager.getThreadGroup() :
Thread.currentThread().getThreadGroup();
AtomicInteger poolCount </span>=<span style="color: #000000;"> POOL_NUMBER.get(threadPoolName);
</span><span style="color: #0000ff;">if</span> (poolCount == <span style="color: #0000ff;">null</span><span style="color: #000000;">) {
namePrefix </span>= threadPoolName + " pool-00-thread-"<span style="color: #000000;">;
} </span><span style="color: #0000ff;">else</span><span style="color: #000000;"> {
namePrefix </span>= threadPoolName + " pool-" + poolCount.getAndIncrement() + "-thread-"<span style="color: #000000;">;
}
}
<span style="color: #0000ff;">public<span style="color: #000000;"> Thread newThread(Runnable runnable) {
Thread thread = <span style="color: #0000ff;">new Thread(group,runnable,namePrefix + threadNumber.getAndIncrement(),0<span style="color: #000000;">);
</span><span style="color: #0000ff;">if</span><span style="color: #000000;"> (thread.isDaemon()) {
thread.setDaemon(</span><span style="color: #0000ff;">false</span><span style="color: #000000;">);
}
</span><span style="color: #0000ff;">if</span> (thread.getPriority() !=<span style="color: #000000;"> Thread.<a href="https://www.jb51.cc/tag/nor/" target="_blank" class="keywords">nor</a>M_PRIORITY) {
thread.setPriority(Thread.<a href="https://www.jb51.cc/tag/nor/" target="_blank" class="keywords">nor</a>M_PRIORITY);
}
</span><span style="color: #0000ff;">return</span><span style="color: #000000;"> thread;
}
}
Obtaining More Detailed information by Using MBean
You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps,by using ThreadInfo.
Highlighter_935995" class="
SyntaxHighlighter ">
ThreadMXBean mxBean =
[] threadIds ==<span style="color: #0000ff;">for<span style="color: #000000;"> (ThreadInfo threadInfo : threadInfos) {
System.out.println(
threadInfo.getThreadName());
System.out.println(
threadInfo.getBlockedCount());
System.out.println(
threadInfo.getBlockedTime());
System.out.println(
threadInfo.getWaitedCount());
System.out.println(
threadInfo.getWaitedTime());
}
nes">
You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo,and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time.
In Conclusion
In this article I was concerned that for developers with a lot of experience in multi-thread programming,this material may be common kNowledge,whereas for less experienced developers,I felt that I was skipping straight to thread dumps,without providing enough background information about the thread activities. This was because of my lack of kNowledge,as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.
相关文章
HashMap是Java中最常用的集合类框架,也是Java语言中非常典型...
在EffectiveJava中的第 36条中建议 用 EnumSet 替代位字段,...
介绍 注解是JDK1.5版本开始引入的一个特性,用于对代码进行说...
介绍 LinkedList同时实现了List接口和Deque接口,也就是说它...
介绍 TreeSet和TreeMap在Java里有着相同的实现,前者仅仅是对...
HashMap为什么线程不安全 put的不安全 由于多线程对HashMap进...