WEB OT linux system admins

Discussion in 'OT Technology' started by ge0, Sep 14, 2009.

  1. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    I am running CentOS 5.3 ..

    I am trying to figure out why my system restarts it self on random days..

    heres something from the messages log

    Code:
    Sep  6 04:03:02 GEODEV syslogd 1.4.1: restart.
    Sep  6 10:15:55 GEODEV dhclient: DHCPREQUEST on eth0 to 192.168.1.1 port 67
    Sep  6 10:15:55 GEODEV dhclient: DHCPACK from 192.168.1.1
    Sep  6 10:15:55 GEODEV dhclient: bound to 192.168.1.121 -- renewal in 247577 seconds.
    Sep  7 03:16:15 GEODEV avahi-daemon[2485]: Invalid query packet.
    Sep  7 03:40:04 GEODEV last message repeated 3 times
    Sep  7 06:41:47 GEODEV last message repeated 3 times
    Sep  7 08:51:33 GEODEV last message repeated 3 times
    Sep  7 09:22:38 GEODEV last message repeated 3 times
    Sep  7 10:44:46 GEODEV last message repeated 3 times
    Sep  7 11:20:51 GEODEV last message repeated 3 times
    Sep  7 12:19:49 GEODEV last message repeated 3 times
    Sep  8 01:23:11 GEODEV last message repeated 3 times
    Sep  8 02:39:42 GEODEV last message repeated 3 times
    Sep  8 03:09:47 GEODEV last message repeated 3 times
    Sep  8 03:11:14 GEODEV last message repeated 3 times
    Sep  8 03:41:27 GEODEV last message repeated 3 times
    Sep  8 04:11:32 GEODEV last message repeated 3 times
    Sep  8 04:41:37 GEODEV last message repeated 3 times
    Sep  8 05:11:42 GEODEV last message repeated 2 times
    Sep  8 05:19:07 GEODEV last message repeated 3 times
    Sep  8 05:32:54 GEODEV last message repeated 3 times
    Sep  8 06:11:22 GEODEV last message repeated 3 times
    Sep  8 07:09:05 GEODEV last message repeated 3 times
    Sep  8 07:36:13 GEODEV last message repeated 3 times
    Sep  8 07:41:03 GEODEV last message repeated 3 times
    Sep  8 08:13:21 GEODEV last message repeated 3 times
    Sep  8 08:43:31 GEODEV last message repeated 3 times
    Sep  8 09:10:23 GEODEV last message repeated 3 times
    Sep  8 09:25:38 GEODEV last message repeated 3 times
    Sep  8 10:58:03 GEODEV last message repeated 3 times
    Sep  8 11:21:12 GEODEV last message repeated 3 times
    Sep  8 11:49:26 GEODEV last message repeated 3 times
    Sep  8 12:03:14 GEODEV last message repeated 3 times
    Sep  8 12:03:14 GEODEV last message repeated 2 times
    Sep  9 07:02:12 GEODEV dhclient: DHCPREQUEST on eth0 to 192.168.1.1 port 67
    Sep  9 07:02:12 GEODEV dhclient: DHCPACK from 192.168.1.1
    Sep  9 07:02:12 GEODEV dhclient: bound to 192.168.1.121 -- renewal in 289088 seconds.
    Sep 10 21:04:10 GEODEV syslogd 1.4.1: restart.
    Sep 10 21:04:10 GEODEV kernel: klogd 1.4.1, log source = /proc/kmsg started.
    Sep 10 21:04:10 GEODEV kernel: Linux version 2.6.18-128.el5 ([EMAIL="[email protected]"][email protected][/EMAIL]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 10:44:23 EST 2009
    
    why do I see these?

    Sep 10 21:04:10 GEODEV syslogd 1.4.1: restart.

    I know for a fact no one has restarted this server, its restarting by itself. Its not overheating. Does ti restart when it fails to retrieve a packet or something?
     
  2. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    Nothing suspicious in there imo. You could try disabling thee avahi shit just for fun, unless you use it
     
  3. hank85

    hank85 sudo shred /dev/sda -f -v -z --iterations=6

    Joined:
    Jul 23, 2008
    Messages:
    4,360
    Likes Received:
    0
    Server is rotating your logs. Upon rotation syslogd is restarted. Nothing to worry about.

    Code:
    man logrotate
     
  4. Supergeek

    Supergeek New Member

    Joined:
    Jan 23, 2007
    Messages:
    1,855
    Likes Received:
    0
    Location:
    Colorado
    Yeah, a real system restart has a lot more shit in it.. hardware detection, all your other services starting, etc.

    DHCP timeout and refresh is normal too.
     
  5. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    well I left the hardware detection out, I figure it wasn't needed. Does CentOS do automatic system updates?
     
  6. hank85

    hank85 sudo shred /dev/sda -f -v -z --iterations=6

    Joined:
    Jul 23, 2008
    Messages:
    4,360
    Likes Received:
    0
    Depends on how you have your system set up. By default, no *nix system will update anything automatically.

    Use yum.
     
  7. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Where would look for failure.. I tried opening the faillog, btu I get a bunch of "@" symbols..
     
  8. RaginBajin

    RaginBajin Have you punched a donkey today?

    Joined:
    Dec 24, 2001
    Messages:
    8,740
    Likes Received:
    0
    Location:
    NoVA
    check your uptime and see if it actually is rebooting or if the log is just rotating. It is a strange time for logs to rotate though.
     
  9. DigiCrime

    DigiCrime If Only!

    Joined:
    Oct 25, 2001
    Messages:
    33,014
    Likes Received:
    101
    Location:
    St. Louis
    centos doesn't reboot on its own no matter what you have going on unless you tell it to. from the command line type

    last

    and see how many times its actually rebooting. type dmesg | more and look thru that stuff and see if you have any start up errors
     
  10. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Code:
    Last login: Tue Sep 15 08:14:36 2009 from ge0laptop
    [[email protected] ~]$ last
    ge0   pts/1        ge0laptop     Tue Sep 15 19:38   still logged in
    ge0   pts/1        ge0laptop     Tue Sep 15 08:14 - 09:10  (00:56)
    ge0   pts/1        ge0laptop     Mon Sep 14 09:10 - 09:39  (00:29)
    ge0   :0                            Thu Sep 10 21:07   still logged in
    ge0   :0                            Thu Sep 10 21:07 - 21:07  (00:00)
    reboot   system boot  2.6.18-128.el5   Thu Sep 10 21:04         (4+22:34)
    ge0   pts/2        ge0laptop     Thu Sep 10 08:52 - 15:09  (06:16)
    ge0   pts/2        ge0laptop     Tue Sep  8 20:01 - 20:02  (00:01)
    ge0   pts/2        ge0laptop     Mon Sep  7 13:18 - 15:30  (02:11)
    ge0   pts/2        geomachine       Mon Sep  7 10:29 - 10:29  (00:00)
    ge0   pts/2        geomachine       Sun Sep  6 12:08 - 12:09  (00:00)
    ge0   pts/2        :0.0             Sat Sep  5 08:11 - 08:11  (00:00)
    ge0   pts/2        geomachine       Fri Sep  4 10:06 - 10:08  (00:01)
    ge0   pts/1        :0.0             Thu Sep  3 12:00 - crash (7+09:04)
    ge0   :0                            Thu Sep  3 11:54 - crash (7+09:09)
    ge0   :0                            Thu Sep  3 11:54 - 11:54  (00:00)
    reboot   system boot  2.6.18-128.el5   Thu Sep  3 11:50         (12+07:47)
    ge0   pts/1        geomachine       Sun Aug 16 17:28 - down   (00:30)
    ge0   :0                            Sun Aug 16 17:24 - down   (00:34)
    ge0   :0                            Sun Aug 16 17:24 - 17:24  (00:00)
    reboot   system boot  2.6.18-128.el5   Sun Aug 16 17:20          (00:38)
    ge0   pts/1        192.168.1.139    Sun Aug 16 10:08 - crash  (07:12)
    ge0   pts/1        192.168.1.139    Sat Aug 15 11:54 - 12:04  (00:10)
    ge0   :0                            Sat Aug 15 11:53 - crash (1+05:27)
    ge0   :0                            Sat Aug 15 11:53 - 11:53  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug 15 11:53         (1+06:05)
    ge0   pts/2        192.168.1.139    Sat Aug 15 12:03 - down   (00:58)
    ge0   pts/1        192.168.1.139    Sat Aug 15 11:41 - down   (01:20)
    ge0   :0                            Sat Aug 15 11:02 - down   (01:59)
    ge0   :0                            Sat Aug 15 11:02 - 11:02  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug 15 11:01          (02:00)
    ge0   pts/1        192.168.1.139    Thu Aug 13 21:52 - crash (1+13:08)
    ge0   :0                            Thu Aug 13 19:19 - crash (1+15:42)
    ge0   :0                            Thu Aug 13 19:19 - 19:19  (00:00)
    reboot   system boot  2.6.18-128.el5   Thu Aug 13 19:18         (1+17:42)
    ge0   pts/1        192.168.1.139    Wed Aug 12 23:24 - 00:05  (00:40)
    ge0   :0                            Wed Aug 12 23:20 - crash  (19:58)
    ge0   :0                            Wed Aug 12 23:20 - 23:20  (00:00)
    reboot   system boot  2.6.18-128.el5   Wed Aug 12 23:19         (2+13:41)
    ge0   pts/2        192.168.1.139    Wed Aug 12 23:12 - down   (00:06)
    ge0   pts/1        192.168.1.139    Wed Aug 12 22:52 - down   (00:25)
    ge0   pts/1        192.168.1.139    Wed Aug 12 22:15 - 22:52  (00:37)
    ge0   pts/1        192.168.1.139    Wed Aug 12 20:10 - 20:32  (00:22)
    ge0   :0                            Wed Aug 12 20:07 - down   (03:11)
    ge0   :0                            Wed Aug 12 20:07 - 20:07  (00:00)
    reboot   system boot  2.6.18-128.el5   Wed Aug 12 20:06          (03:11)
    ge0   pts/1        192.168.1.139    Mon Aug 10 20:23 - 20:23  (00:00)
    ge0   pts/1        192.168.1.139    Sat Aug  8 15:20 - 18:31  (03:11)
    ge0   :0                            Sat Aug  8 14:57 - crash (4+05:09)
    ge0   :0                            Sat Aug  8 14:57 - 14:57  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  8 14:57         (4+08:21)
    ge0   pts/1        192.168.1.139    Wed Aug  5 20:06 - crash (2+18:50)
    ge0   :0                            Tue Aug  4 19:40 - crash (3+19:16)
    ge0   :0                            Tue Aug  4 19:40 - 19:40  (00:00)
    reboot   system boot  2.6.18-128.el5   Tue Aug  4 19:39         (8+03:38)
    ge0   pts/2        192.168.1.129    Tue Aug  4 19:32 - 19:38  (00:06)
    ge0   pts/2        192.168.1.129    Tue Aug  4 19:23 - 19:24  (00:00)
    ge0   pts/1        192.168.1.139    Mon Aug  3 00:32 - down  (1+19:05)
    ge0   :0                            Mon Aug  3 00:28 - down  (1+19:09)
    ge0   :0                            Mon Aug  3 00:28 - 00:28  (00:00)
    reboot   system boot  2.6.18-128.el5   Mon Aug  3 00:28         (1+19:10)
    ge0   pts/1        192.168.1.139    Sun Aug  2 15:38 - down   (00:02)
    ge0   pts/2        :0.0             Sun Aug  2 15:36 - 15:38  (00:01)
    ge0   pts/1        :0.0             Sat Aug  1 22:28 - 15:38  (17:09)
    ge0   :0                            Sat Aug  1 21:10 - down   (18:30)
    ge0   :0                            Sat Aug  1 21:10 - 21:10  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 21:09          (18:31)
    ge0   :0                            Sat Aug  1 19:39 - crash  (01:30)
    ge0   :0                            Sat Aug  1 19:39 - 19:39  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 19:37          (20:03)
    ge0   :0                            Sat Aug  1 19:18 - 19:22  (00:03)
    ge0   :0                            Sat Aug  1 19:18 - 19:18  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 19:18          (00:03)
    ge0   pts/1        192.168.1.139    Sat Aug  1 18:21 - down   (00:04)
    ge0   :0                            Sat Aug  1 18:20 - down   (00:05)
    ge0   :0                            Sat Aug  1 18:20 - 18:20  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 18:19          (00:05)
    ge0   pts/1        :0.0             Sat Aug  1 18:11 - 18:18  (00:06)
    ge0   :0                            Sat Aug  1 18:06 - 18:18  (00:11)
    ge0   :0                            Sat Aug  1 18:06 - 18:06  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 18:04          (00:13)
    ge0   pts/1        :0.0             Sat Aug  1 17:51 - 17:54  (00:02)
    ge0   :0                            Sat Aug  1 17:51 - 17:54  (00:03)
    ge0   :0                            Sat Aug  1 17:51 - 17:51  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 17:50          (00:03)
    ge0   pts/4        :2.0             Sat Aug  1 17:07 - 17:09  (00:01)
    ge0   pts/3        :1.0             Sat Aug  1 17:04 - 17:09  (00:04)
    ge0   pts/2        192.168.1.139    Sat Aug  1 16:58 - down   (00:11)
    ge0   pts/1        :0.0             Sat Aug  1 16:48 - 17:09  (00:21)
    ge0   pts/0        :0.0             Sat Aug  1 16:33 - 17:09  (00:35)
    ge0   :0                            Sat Aug  1 16:09 - 17:09  (00:59)
    ge0   :0                            Sat Aug  1 16:09 - 16:09  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 16:07          (01:01)
    ge0   :0                            Sat Aug  1 16:04 - 16:05  (00:01)
    ge0   :0                            Sat Aug  1 16:04 - 16:04  (00:00)
    ge0   pts/1        :0.0             Sat Aug  1 16:01 - 16:03  (00:02)
    ge0   :0                            Sat Aug  1 15:53 - 16:03  (00:10)
    ge0   :0                            Sat Aug  1 15:53 - 15:53  (00:00)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 15:52          (00:12)
    reboot   system boot  2.6.18-128.el5   Sat Aug  1 15:41          (00:09)
    
    I didn;t find anything suspicious in the dmesg
     
  11. DigiCrime

    DigiCrime If Only!

    Joined:
    Oct 25, 2001
    Messages:
    33,014
    Likes Received:
    101
    Location:
    St. Louis
    edit: read that wrong. This a server/desktop something you own (colo) or dedicated ?
     
  12. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Dell poweredge I own sitting in my basement.
     
  13. Supergeek

    Supergeek New Member

    Joined:
    Jan 23, 2007
    Messages:
    1,855
    Likes Received:
    0
    Location:
    Colorado
    If you can't read "last" output, run 'uptime' and it'll tell you exactly how long your system has been running WITHOUT A REBOOT OR CRASH!

     
  14. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Code:
    [[email protected] samba]# uptime
     07:45:22 up 6 days, 10:42,  2 users,  load average: 0.00, 0.00, 0.00
    
     
  15. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Code:
    george   pts/1        geomanlaptop     Sat Sep 19 07:13   still logged in
    george   :0                            Fri Sep 18 22:21   still logged in
    george   :0                            Fri Sep 18 22:21 - 22:21  (00:00)
    reboot   system boot  2.6.18-128.el5   Fri Sep 18 22:18          (10:15)
    george   pts/1        geomanlaptop     Wed Sep 16 08:39 - crash (2+13:38)
    
     
  16. crazybenf

    crazybenf Active Member

    Joined:
    Nov 14, 2001
    Messages:
    15,578
    Likes Received:
    3
    its been up for almost a week...
     
  17. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    why would it restart itself tho after a week?
     
  18. crazybenf

    crazybenf Active Member

    Joined:
    Nov 14, 2001
    Messages:
    15,578
    Likes Received:
    3
    shitty power? centos doesnt just restart on its own.
     
  19. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    heres what werid, My other poweredge has been going strong for 2 months without a restart... PLugged into the same outlet

    werid, I will look into that power problem some more.
     
  20. FormulaLS1

    FormulaLS1 Member

    Joined:
    Nov 14, 2006
    Messages:
    519
    Likes Received:
    0
    You can probably change some syslog settings and make the logs more verbose...
     
  21. DigiCrime

    DigiCrime If Only!

    Joined:
    Oct 25, 2001
    Messages:
    33,014
    Likes Received:
    101
    Location:
    St. Louis
    This, forgot about that. Your system is restarting on its own for some other reason probably hardware or someone else has access to your system and exploiting it. Unlike windows, linux won't restart on its own without a manual restart or a power trip.

    I would setup your logging situation to get a more defined output ie higher debugging level. If it helps any move your old logs off somewhere and create new ones so you can get a fresh start on logs. Rule out hardware failure using standard diagnostic testing (memory and drive testing/benckmark) Power supplies can be tested to but you need a testing tool for this to see if it is putting out the right voltage.

    Process of elimination at this point. Nothing comes to my mind what would cause it except an electrical issue.
     
  22. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY
    Electrical it was. Had them on an older powersurge thing.. took it off that and its been fine for a month..

    Thanks Guys:)
     
  23. Ricky

    Ricky █▄ █▄█ █▄ ▀█▄

    Joined:
    Jun 17, 2005
    Messages:
    38,766
    Likes Received:
    6
    some dude at your hosting company is fucking with you :squint:
     
  24. jzakhar

    jzakhar Custom User Title

    Joined:
    Jan 26, 2005
    Messages:
    340
    Likes Received:
    0
    Location:
    San Diego, California
    capture the output of dmesg as well

    # dmesg > sometextfileillreadlater.txt

    or

    # dmesg | less

    and page through it

    it frequently will have messages if you're having hardware problems
     
  25. ge0

    ge0 New Member

    Joined:
    Oct 31, 2005
    Messages:
    8,398
    Likes Received:
    0
    Location:
    JERSEY

    I am the hosting company.. I got few poweredges running in my basement.:mamoru:
     

Share This Page