Tuesday, February 8, 2011

Useful UNIX commands for WebSphere Administrator

Useful UNIX commands for WebSphere Administrator
 

These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.

Performance related Commands to monitor system performance
-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU

 

 

%CPU   PID USER     COMMAND
 71.7 31237 root     /cust/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere

/AppServer/java/jre/lib/ext/ibmext.jar
-Dwas.status.socket=54859 -classpath

/cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer






2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

 

 

%CPU   PID USER     COMMAND
2.1  6375 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
 0.3 13886 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
 0.2 17271 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
 0.2 17871 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
 0.2 17713 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
 0.1 18499 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
 0.1 17504 root     /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer





3)Find cpu usage
mpstat -P ALL or mpstat

 

 

Linux 2.4.21-27.ELsmp (machine1.www.com)   02/02/2008

11:55:00 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:55:00 AM  all    1.51    0.00    0.41    0.04    0.01    0.09   97.94    184.92
11:55:00 AM    0    1.50    0.00    0.41    0.04    0.01    0.11   97.93    123.76
11:55:00 AM    1    1.51    0.00    0.42    0.04    0.00    0.07   97.95     61.16





4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top

 

 


12:00:20  up 118 days, 15:16,  2 users,  load average: 0.07, 0.02, 0.00
572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    2.7%    0.0%    1.1%   0.0%     0.0%    0.0%   95.9%
           cpu00    1.9%    0.0%    0.3%   0.0%     0.0%    0.0%   97.6%
           cpu01    3.5%    0.0%    1.9%   0.1%     0.0%    0.0%   94.2%
Mem:  11819592k av, 5088092k used, 6731500k free,       0k shrd,  295304k buff
      3998132k active,             682312k inactive
Swap: 2048276k av,       0k used, 2048276k free                 1258592k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
17719 root      16   0  306M 306M  3992 S     0.7  2.6  22:14   1 java
19358 root      16   0  2532 2532   896 R     0.5  0.0   0:00   1 top
17771 root      15   0  306M 306M  3992 S     0.4  2.6   3:48   0 java
18058 root      16   0  458M 458M  4032 S     0.1  3.9  12:55   0 java
    1 root      15   0   512  512   452 S     0.0  0.0   1:19   0 init
    2 root      RT   0     0    0     0 SW    0.0  0.0   0:00   0 migration/0





5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

 

 

Linux 2.4.21-27.ELsmp (machine1.web.com)   02/02/2008

12:04:34 PM       CPU     %user     %nice   %system   %iowait     %idle
12:04:36 PM       all      0.25      0.00      0.00      0.00     99.75
12:04:38 PM       all      0.00      0.00      0.50      0.25     99.25
12:04:40 PM       all      7.00      0.00      7.25      0.00     85.75
12:04:42 PM       all     21.05      0.00      1.50      0.00     77.44
12:04:44 PM       all      3.00      0.00      0.50      0.00     96.50
Average:          all      6.26      0.00      1.95      0.05     91.74






Disk Usage related Commands to monitor Disk space
--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T

 

 


Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    2.9G  785M  2.0G  29% /
/dev/sda7     ext3     57G  9.3G   45G  18% /cust
/dev/sda6     ext3    2.0G   82M  1.8G   5% /lc
none         tmpfs    5.7G     0  5.7G   0% /dev/shm
/dev/sda5     ext3    2.0G   33M  1.8G   2% /tmp
/dev/sda3     ext3    2.0G   64M  1.8G   4% /var





2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total

 

 


bash-2.05b# du -ch | grep total
6.3G    total






Network related Commands
-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.

The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com


a) dig <ip address or hostname>

 

 


bash-2.05b# dig www.google.com

; <<>> DiG 9.2.4 <<>> www.google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         585119  IN      CNAME   www.l.google.com.
www.l.google.com.       69      IN      A       216.239.51.99
www.l.google.com.       69      IN      A       216.239.51.104

;; AUTHORITY SECTION:
l.google.com.           60734   IN      NS      g.l.google.com.
l.google.com.           60734   IN      NS      b.l.google.com.
l.google.com.           60734   IN      NS      f.l.google.com.
l.google.com.           60734   IN      NS      c.l.google.com.
l.google.com.           60734   IN      NS      a.l.google.com.
l.google.com.           60734   IN      NS      e.l.google.com.
l.google.com.           60734   IN      NS      d.l.google.com.

;; Query time: 2 msec
;; SERVER: 10.5.50.184#53(10.5.50.184)
;; WHEN: Sun Feb  3 09:51:46 2008
;; MSG SIZE  rcvd: 196





b) host <ip address or hostname>

 

 


bash-2.05b# host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 216.239.51.104
www.l.google.com has address 216.239.51.99





c) nslookup <ip address or hostname>

 

 


bash-2.05b# nslookup www.google.com
Server:         10.5.50.184
Address:        10.5.50.184#53

Non-authoritative answer:
www.google.com  canonical name = www.l.google.com.
Name:   www.l.google.com
Address: 216.239.51.99
Name:   www.l.google.com
Address: 216.239.51.104





2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback.


infconfig -a

 

 


bash-2.05b# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:11:43:32:49:99 
          inet addr:10.2.150.74  Bcast:10.5.50.127  Mask:255.255.255.192
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0
          TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3168219417 (3021.4 Mb)  TX bytes:3724305382 (3551.7 Mb)
          Interrupt:17

eth1      Link encap:Ethernet  HWaddr 00:11:43:32:49:9A 
          inet addr:10.5.154.138  Bcast:10.5.54.127  Mask:255.255.255.128
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:18

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0
          TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1454141744 (1386.7 Mb)  TX bytes:1454141744 (1386.7 Mb)






3) To find more information about a network interface, the below example shows detail information about interface eth0.


bash-2.05b# ethtool eth0

 

 


Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes





File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation.

find . -name '*.xml' -print | xargs grep 'hostName' /dev/null

 

 


bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.xml' -print | xargs grep 'hostName' /dev/null  | more
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverNode01/serverindex.xml:<serverindex:
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointRefs="NamedEndPoint_1130375480667 NamedE
ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPoint_1130375480671 NamedEndPoint_113037548067
2 NamedEndPoint_1130375480673">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverCellManager01/serverindex.xml:  <ser
verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_2" hostName="portalserver.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2011indchn2Node01/serverindex.xml:<serverindex:
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointRefs="NamedEndPoint_1130375717654 NamedE
ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPoint_1130375717658 NamedEndPoint_113037571765
9 NamedEndPoint_1130375717660">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2UnNode01/serverindex.xml:<serverinde
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2UnNode01/serverindex.xml:<serverinde
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serverindex:ServerI
ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.0/serv
erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode01Cell/nodes/portalserverNode01/serverindex.x
ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/app
server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serveri
ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/sche
mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">





2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in size.

find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

 

 


bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-rw-r--r--  
 1 root     root      1047553 Oct 26  2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/tranlog/log1
-rw-r--r--    1 root     root      1047553 Oct 26  2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log2
-rw-r--r--    1 root     root      1047553 Oct 26  2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log1





3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload.

To pack on the fly...

tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz

To unpack on the fly..
gunzip < FILE.tar.gz | tar -xvf -

To pack on the fly and email the file to IBM Support
tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " WebSphere_Support@mainz.ibm.com

To find selective files and tar it up on the fly...
find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar


4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size.


ls -l | tee > filename
xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log

5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime)


find /usr/IBM -mmin -20
find /usr/IBM -mtime -20

 

 


/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs  





6) Find a memory usage of the system.


free
cat /proc/meminfo

 

 


bash-2.05b# free
             total       used       free     shared    buffers     cached
Mem:      11819592    4938416    6881176          0     298148    1440200
-/+ buffers/cache:    3200068    8619524
Swap:      2048276          0    2048276

bash-2.05b# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  12103262208 5056954368 7046307840        0 305303552 1474797568
Swap: 2097434624        0 2097434624
MemTotal:     11819592 kB
MemFree:       6881160 kB
MemShared:           0 kB
Buffers:        298148 kB
Cached:        1440232 kB
SwapCached:          0 kB
Active:        3718616 kB
ActiveAnon:    2968776 kB
ActiveCache:    749840 kB
Inact_dirty:    789328 kB
Inact_laundry:  237596 kB
Inact_clean:         0 kB
Inact_target:   949108 kB
HighTotal:    11140992 kB
HighFree:      6681628 kB
LowTotal:       678600 kB
LowFree:        199532 kB
SwapTotal:     2048276 kB
SwapFree:      2048276 kB
Committed_AS:  6039100 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB





4) To repeat a command contimuously.

watch -d=10 -n 1 "netstat -a"

 

 


Every 1s: netstat -a| grep ESTABLISHED                                                                      Sun Feb  3 21:48:04 2008

tcp        0      0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 ESTABLISHED
tcp        0      0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 ESTABLISHED
tcp        0      0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 ESTABLISHED
tcp        0      0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 ESTABLISHED
tcp        0      0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545 ESTABLISHED