Enter Solaris 8 brandz zones. Being able to run legacy applications in a Solaris 8 brandz generally gets you around most of the issues in the remaining apps that do stupid things. However, there’s a couple of quirks with a zone that can introduce odd behaviour in an application.

Progress for example.…… Read On.

The issue

An old legacy Progress database was sitting on an old legacy server which was so EOL that it would make even Bill cringe… or laugh… Running Solaris 6 was not an option, but running anything else, (even Solaris 9), would be adventurous to say the least. Can I say legacy again?

The solution

So first try a brandz Solaris 8. Configured, installed, and booted a brand spanking new zone. Copy the data over. Attempt a start of the Progress database.

10:03:21 BROKER  0: Multi-user session begin. (333)
10:03:24 BROKER  0: Begin Physical Redo Phase at 13312 . (5326)
10:03:30 BROKER  0: Physical Redo Phase Completed at blk 13743 off 2457 upd 2636. (7161)
10:03:31 BROKER  0: Started for mfglive using TCP, pid 20533. (5644)
10:04:02 BROKER  0: SYSTEM ERROR: Unable to kill parent process, errno= 4. (1680)
10:04:02 BROKER  0: SYSTEM ERROR: The broker is exiting unexpectedly, beginning Abnormal Shutdown. (5292)
10:04:02 BROKER  0: drexit: Initiating Abnormal Shutdown
10:04:02 BROKER  0: ** Save file named core for analysis by Progress Software Corporation. (439)
10:04:02 BROKER  0: Begin ABNORMAL shutdown code 2 (2249)
10:04:04 BROKER   : Removed shared memory with segment_id: 41
10:04:04 BROKER   : Multi-user session end. (334)

Nada. Mmmmm. Seems to start up OK, but then crashes in a screaming heap a short while later.

Checking shared libraries seemed to be OK. Nothing missing or odd.

root % ldd -r /mfgpro/dlc91c/bin/_mprosrv
        /usr/lib/secure/s8_preload.so.1
        libsocket.so.1 =>        /usr/lib/libsocket.so.1
        libnsl.so.1 =>   /usr/lib/libnsl.so.1
        libintl.so.1 =>  /usr/lib/libintl.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        libm.so.1 =>     /usr/lib/libm.so.1
        libthread.so.1 =>        /usr/lib/libthread.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libmp.so.2 =>    /usr/lib/libmp.so.2
        /usr/platform/SUNW,Sun-Fire-V490/lib/libc_psr.so.1

So, I thought I’d truss, (the old friend), a few processes to see what was going on. Chucking the below in various scripts, and comparing the working server to the non-working server. (This produces VERY NOISY output, but it’s great for diagnosing what’s going on.)

truss -feal -vall -xall -rall -wall -sall -mall -o /tmp/logfile exec_process

If you look at the appendices to this post, I’ve added the relevant, sections of the truss. You may notice an interesting thing. The working server roughly follows this process, (in the snippet):
– Bind to 0.0.0.0 port 3500.
– fstat64() filehandles.
– Yadda, yadda, write output to STDOUT.
– getpid() — find parent process, (24012).
– kill –TERM 24012
– wait()
– Parent — ALRM signal
– Parent — die.
– Child — Context switch.
– getpid() — find parent process, (1). OK!

Now this is a little odd. It would seem that some developer thought that it’d be a damn good idea to re-invent the old “I’m going to fork a child process in the background” API calls, and chuck in their own.

They are checking the parent process, killing it, and rechecking to see if it’s now 1, (which is sysvinit), then it progresses on.

Now, the non-working server.
– Bind to 0.0.0.0 port 3500.
– fstat64() filehandles.
– Yadda, yadda, write output to STDOUT.
– getpid() — find parent process, (21267).
– kill –TERM 21267
– wait()
– Parent — ALRM signal <-IMPORTANT
– Parent — no die.
– getpid() — find parent process, (21267).
– kill –TERM 21267
– wait()
– Parent — ALRM signal <-IMPORTANT
– Parent — no die.
Ad naseum.….

A quick process check:

     UID   PID  PPID  C    STIME TTY      TIME CMD
    root  2260  1417  0   May 27 ?        0:09 /etc/init
    root  1417  1417  0   May 27 ?        0:00 zsched
    root  2960  1417  0   May 27 ?        0:00 /usr/lib/picl/picld

Hey! Of course you’re not gonna be able to kill zsched! You see in a Solaris zone the ‘root’ process is called ‘zsched’, and it will never ever be 1 at all, and neither will init for that matter. Not only that you’re not ever going to be able to kill the zsched process from within the zone as it’s protected within the global zone memory space.

Well, it’s too late to kick the developer up the bum. So time to hack.

The hack

Preloading shared libs has been around since I was a Mars Bar in my Dad’s top pocket, (well almost). So it should be a simple case of replacing any getpid() API call with my own. Simple enough, done it before.

See the appendix for the getpidhack.c file that I used. This is a shared lib that replaces both getppid() and getpid(), (for good measure, but not really needed in my case), API calls. It will return the PID 1, when _getppid() returns the same PID as the environment variable PIDHACK.
Note that I call _getppid() which allows me to fetch the REAL PID so I can return it when they don’t match.

So, to use it. First build it:

gcc -fPIC -c getpidhack.c
gld getpidhack.o -L/usr/lib -lc -G -assert pure-text -o /usr/local/lib/libgetpidhack.so

Set an environment variable called PIDHACK to the currently running zsched process:

HACKPID=`ps -e | awk '/zsched/{print$1}'`
export HACKPID

Then set your LD_PRELOAD variable to point to the shared lib:

LD_PRELOAD="/usr/local/lib/libgetpidhack.so"
export LD_PRELOAD

After this the Progress database just started. Woot! The DB can now run on hardware that it would never in it’s life be able to run on under a brandz zone. Another EOL server to chuck out the computer room window to see what happens.

BTW, if you are having issues with this, you can run a DTrace to see if you’re calling the real getpid(), or the getpidhack version. Create the following script, and run it with one argument, (the PID of your zsched). Every time you hit the getppid() API call it’ll register — this means that you are not replacing the API call and your shared lib preload isn’t working.

#!/usr/sbin/dtrace -s

syscall::getpid:return
/ppid == $1/
{
        printf("%d %d", ppid, pid);
}

Appendix 1 — getpidhack.c

#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>

pid_t getppid(void)
{
        pid_t hackppid;
        pid_t realppid;

        // hackpid - the PID of the zsched process.
        hackppid = atol(getenv("HACKPID"));

        // realppid - the real PID of the parent process.
        realppid = _getppid();

        // If the realppid equals hackppid, then we want to reset it to 1.
        if (realppid == hackppid)
        {
                return((pid_t) 1);
        }

        // Else return whatever PID we were given.
        return(realppid);
}

pid_t getpid(void)
{
        pid_t hackpid;
        pid_t realpid;

        // hackpid - the PID of the zsched process.
        hackpid = atol(getenv("HACKPID"));

        // realpid - the real PID of the parent process.
        realpid = _getpid();

        // If the realpid equals hackpid, then we want to reset it to 1.
        if (realpid == hackpid)
        {
                return((pid_t) 1);
        }

        // Else return whatever PID we were given.
        return(realpid);
}

Appendix 2 — The working server

24013/1:        setsockopt(76, 65535, 4, 0xEFFFF7FC, 4)         = 0
24013/1:        setsockopt(76, 65535, 8, 0xEFFFF7FC, 4)         = 0
24013/1:        bind(76, 0x0015DCA8, 16)                        = 0
24013/1:                name = 0.0.0.0/3500
24013/1:        listen(76, 10)                                  = 0
24013/1:        getpid()                                        = 24013 [24012]
24013/1:        lseek(3, 457164, 0)                             = 457164
24013/1:        read(3, 0xEFFFF66C, 81)                         = 81
24013/1:           % L S t a r t e d   f o r   % s   u s i n g   % s ,   p i d  %
24013/1:           l .   ( 5 6 4 4 )\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 \0
24013/1:          \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
24013/1:        time()                                          = 1252542477
24013/1:        write(9, 0xEFFFF154, 69)                        = 69
24013/1:           1 0 : 2 7 : 5 7   B R O K E R     0 :   S t a r t e d   f o r
24013/1:           m f g l i v e   u s i n g   T C P ,   p i d   2 4 0 1 3 .   ( 5
24013/1:           6 4 4 )\n
24013/1:        fstat64(0, 0xEFFFF058)                          = 0
24013/1:            d=0x01540000 i=17139 m=0020620 l=1  u=0     g=7     rdev=0x0 0600005
24013/1:                at = Sep 10 10:27:46 EST 2009  [ 1252542466 ]
24013/1:                mt = Sep 10 10:27:56 EST 2009  [ 1252542476 ]
24013/1:                ct = Sep 10 09:39:31 EST 2009  [ 1252539571 ]
24013/1:            bsz=8192  blks=0     fs=ufs
24013/1:        fstat64(1, 0xEFFFF058)                          = 0
24013/1:            d=0x01540000 i=17139 m=0020620 l=1  u=0     g=7     rdev=0x0 0600005
24013/1:                at = Sep 10 10:27:46 EST 2009  [ 1252542466 ]
24013/1:                mt = Sep 10 10:27:56 EST 2009  [ 1252542476 ]
24013/1:                ct = Sep 10 09:39:31 EST 2009  [ 1252539571 ]
24013/1:            bsz=8192  blks=0     fs=ufs
24013/1:        fstat64(2, 0xEFFFF058)                          = 0
24013/1:            d=0x01540000 i=17139 m=0020620 l=1  u=0     g=7     rdev=0x0 0600005
24013/1:                at = Sep 10 10:27:46 EST 2009  [ 1252542466 ]
24013/1:                mt = Sep 10 10:27:56 EST 2009  [ 1252542476 ]
24013/1:                ct = Sep 10 09:39:31 EST 2009  [ 1252539571 ]
24013/1:            bsz=8192  blks=0     fs=ufs
24013/1:        write(1, 0xEFFFF154, 69)                        = 69
24013/1:           1 0 : 2 7 : 5 7   B R O K E R     0 :   S t a r t e d   f o r
24013/1:           m f g l i v e   u s i n g   T C P ,   p i d   2 4 0 1 3 .   ( 5
24013/1:           6 4 4 )\n
24013/1:        pwrite64(7, 0xEFFFF908, 4, 0)                   = 4
24013/1:             0xEFFFF908: "\0\0\002"
24013/1:        close(7)                                        = 0
24013/1:        getpid()                                        = 24013 [24012]
24013/1:        getpid()                                        = 24013 [24012]
24013/1:        kill(24012, 0x0000000F)                         = 0
24012/2:        signotifywait()                                 = 15
24013/1:        lwp_alarm(1)                                    = 0
24012/2:            Incurred fault #11, FLTPAGE  %pc = 0xEF5CDC80  addr = 0xEF67 0F48
24012/2:            Incurred fault #11, FLTPAGE  %pc = 0xEF64A3C8  addr = 0xEF66 FA68
24012/2:        lwp_sigredirect(1, 0x0000000F)                  = 0
24012/1:            Received signal #15, SIGTERM, in wait() [caught]
24012/1:              siginfo: SIGTERM pid=24013 uid=0
24012/1:        wait()                                          Err#4 EINTR
24012/1:        sigprocmask(3, 0xEF667DF8, 0x00000000)          = 0
24012/1:                 set = 0 0 0 0
24012/1:            Incurred fault #11, FLTPAGE  %pc = 0x0004593C  addr = 0x0013 10F4
24012/1:            Incurred fault #11, FLTPAGE  %pc = 0x000459A8  addr = 0x0015 3AEC
24012/1:        _exit(0)
24013/1:            Received signal #14, SIGALRM, in pause() [caught]
24013/1:        pause()                                         Err#4 EINTR
24013/1:            Incurred fault #11, FLTPAGE  %pc = 0xEF5CDCE8  addr = 0xEF5C DCE8
24013/1:        sigprocmask(3, 0xEF667DF8, 0x00000000)          = 0
24013/1:                 set = 0 0 0 0
24013/1:        context(1, 0xEFFFF240)
24013/1:        getpid()                                        = 24013 [1]
24013/1:        close(0)                                        = 0
24013/1:        open(0xEFFFF920, 0)                             = 0
24013/1:             0xEFFFF920: "/qaddb/eblive/mfglive.lg"
24013/1:        close(1)                                        = 0
24013/1:        open(0xEFFFF920, 0)                             = 1
24013/1:             0xEFFFF920: "/qaddb/eblive/mfglive.lg"
24013/1:        close(2)                                        = 0
24013/1:        open(0xEFFFF920, 0)                             = 2
24013/1:             0xEFFFF920: "/qaddb/eblive/mfglive.lg"
24013/1:        lseek(3, 342954, 0)                             = 342954
24013/1:        read(3, 0xEFFFF570, 81)                         = 81
24013/1:           % L P R O G R E S S   V e r s i o n   % s   o n   % s .   ( 4 2
24013/1:           3 4 )\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 \0
24013/1:          \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
24013/1:        time()                                          = 1252542478
24013/1:        write(9, 0xEFFFF05C, 61)                        = 61
24013/1:           1 0 : 2 7 : 5 8   B R O K E R     0 :   P R O G R E S S   V e r
24013/1:           s i o n   9 . 1 C   o n   S O L A R I S .   ( 4 2 3 4 )\n
24013/1:        fstat64(0, 0xEFFFEF60)                          = 0
24013/1:            d=0x01540002 i=68332 m=0100644 l=1  u=0     g=1     sz=97192 88
24013/1:                at = Sep 10 03:01:28 EST 2009  [ 1252515688 ]
24013/1:                mt = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:                ct = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:            bsz=8192  blks=19008 fs=ufs
24013/1:        fstat64(1, 0xEFFFEF60)                          = 0
24013/1:            d=0x01540002 i=68332 m=0100644 l=1  u=0     g=1     sz=97192 88
24013/1:                at = Sep 10 03:01:28 EST 2009  [ 1252515688 ]
24013/1:                mt = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:                ct = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:            bsz=8192  blks=19008 fs=ufs
24013/1:        fstat64(2, 0xEFFFEF60)                          = 0
24013/1:            d=0x01540002 i=68332 m=0100644 l=1  u=0     g=1     sz=97192 88
24013/1:                at = Sep 10 03:01:28 EST 2009  [ 1252515688 ]
24013/1:                mt = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:                ct = Sep 10 10:27:58 EST 2009  [ 1252542478 ]
24013/1:            bsz=8192  blks=19008 fs=ufs
24013/1:        write(1, 0xEFFFF05C, 61)                        Err#9 EBADF
24013/1:           1 0 : 2 7 : 5 8   B R O K E R     0 :   P R O G R E S S   V e r
24013/1:           s i o n   9 . 1 C   o n   S O L A R I S .   ( 4 2 3 4 )\n
24013/1:        lseek(3, 346761, 0)                             = 346761
24013/1:        read(3, 0xEFFFF570, 81)                         = 81
24013/1:           % L S e r v e r   s t a r t e d   b y   % s   o n   % s .   ( 4
24013/1:           2 8 1 )\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 \0
24013/1:          \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
24013/1:        time()                                          = 1252542478
24013/1:        write(9, 0xEFFFF05C, 65)                        = 65
24013/1:           1 0 : 2 7 : 5 8   B R O K E R     0 :   S e r v e r   s t a r t
24013/1:           e d   b y   r o o t   o n   / d e v / p t s / 5 .   ( 4 2 8 1)
24013/1:          \n

Appendix 3 — The non-working server

21268/1:        setsockopt(76, 65535, 4, 0xFFBFF9AC, 4, 1)      = 0
21268/1:        setsockopt(76, 65535, 8, 0xFFBFF9AC, 4, 1)      = 0
21268/1:        bind(76, 0x0015DCA8, 16, 3)                     = 0
21268/1:                AF_INET  name = 0.0.0.0  port = 3500
21268/1:        listen(76, 10, 1)                               = 0
21268/1:        getpid()                                        = 21268 [21267]
21268/1:        lseek(3, 457164, 0)                             = 457164
21268/1:        read(3, 0xFFBFF81C, 81)                         = 81
21268/1:           % L S t a r t e d   f o r   % s   u s i n g   % s ,   p i d   %
21268/1:           l .   ( 5 6 4 4 )\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
21268/1:          \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
21268/1:        time()                                          = 1252542489
21268/1:        write(9, 0xFFBFF304, 69)                        = 69
21268/1:           1 0 : 2 8 : 0 9   B R O K E R     0 :   S t a r t e d   f o r
21268/1:           m f g l i v e   u s i n g   T C P ,   p i d   2 1 2 6 8 .   ( 5
21268/1:           6 4 4 )\n
21268/1:        fstat64(0, 0xFFBFF208)                          = 0
21268/1:            d=0x05980009 i=4426329 m=0020620 l=1  u=0     g=7     rdev=0x00600011
21268/1:                at = Sep 10 10:27:57 EST 2009  [ 1252542477 ]
21268/1:                mt = Sep 10 10:28:09 EST 2009  [ 1252542489 ]
21268/1:                ct = Sep 10 09:44:02 EST 2009  [ 1252539842 ]
21268/1:            bsz=8192  blks=0     fs=lofs
21268/1:        fstat64(1, 0xFFBFF208)                          = 0
21268/1:            d=0x05980009 i=4426329 m=0020620 l=1  u=0     g=7     rdev=0x00600011
21268/1:                at = Sep 10 10:27:57 EST 2009  [ 1252542477 ]
21268/1:                mt = Sep 10 10:28:09 EST 2009  [ 1252542489 ]
21268/1:                ct = Sep 10 09:44:02 EST 2009  [ 1252539842 ]
21268/1:            bsz=8192  blks=0     fs=lofs
21268/1:        fstat64(2, 0xFFBFF208)                          = 0
21268/1:            d=0x05980009 i=4426329 m=0020620 l=1  u=0     g=7     rdev=0x00600011
21268/1:                at = Sep 10 10:27:57 EST 2009  [ 1252542477 ]
21268/1:                mt = Sep 10 10:28:09 EST 2009  [ 1252542489 ]
21268/1:                ct = Sep 10 09:44:02 EST 2009  [ 1252539842 ]
21268/1:            bsz=8192  blks=0     fs=lofs
21268/1:        write(1, 0xFFBFF304, 69)                        = 69
21268/1:           1 0 : 2 8 : 0 9   B R O K E R     0 :   S t a r t e d   f o r
21268/1:           m f g l i v e   u s i n g   T C P ,   p i d   2 1 2 6 8 .   ( 5
21268/1:           6 4 4 )\n
21268/1:        pwrite64(7, 0xFFBFFAB8, 4, 0)                   = 4
21268/1:             0xFFBFFAB8: "\0\0\002"
21268/1:        close(7)                                        = 0
21268/1:        getpid()                                        = 21268 [21267]
21268/1:        kill(21267, 0x0000000F)                         = 0
21267/1:            Received signal #15, SIGTERM, in wait() [caught]
21267/1:              siginfo: SIG#0
21268/1:            Incurred fault #11, FLTPAGE  %pc = 0x000C3048  addr = 0x58900137234
21267/1:        wait()                                          Err#4 EINTR
21268/1:        alarm(1)                                        = 0
21267/1:        lwp_sigtimedwait(0xFFBFF570, 0xFFBFF498, 0x00000010) = 0
21267/1:                sigmask = 0 0 0 0
21267/1:              siginfo: SIG#0
21267/1:        lwp_sigtimedwait(0xFFBFF488, 0xFFBFF570, 0x00000010) = 0
21267/1:                sigmask = 0 0 0 0
21267/1:              siginfo: SIG#0
21267/1:        lwp_sigtimedwait(0xFFBFF484, 0xFFBFF2B8, 0x00000010) = 0
21267/1:                sigmask = 0x00004000 0 0 0
21267/1:              siginfo: SIG#16384
21267/1:        lwp_sigtimedwait(0xFFBFF2A8, 0xFFBFF330, 0x00000010) = 0
21267/1:                sigmask = 0x00004000 0 0 0
21267/1:              siginfo: SIG#16384
21267/1:        sigprocmask(3, 0xFFBFF330, 0x00000000)          = 0
21267/1:                 set = 0x00004000 0 0 0
21267/1:            Incurred fault #11, FLTPAGE  %pc = 0x0004593C  addr = 0x589001310F4
21267/1:            Incurred fault #11, FLTPAGE  %pc = 0x000459A8  addr = 0x58900153AEC
21267/1:        _exit(0)
21268/1:            Received signal #14, SIGALRM, in pause() [caught]
21268/1:              siginfo: SIG#0
21268/1:        pause()                                         Err#4 EINTR
21268/1:        lwp_sigtimedwait(0xFFBFF698, 0xFFBFF5C0, 0x00000010) = 0
21268/1:                sigmask = 0 0 0 0
21268/1:              siginfo: SIG#0
21268/1:        lwp_sigtimedwait(0xFFBFF5B0, 0xFFBFF698, 0x00000010) = 0
21268/1:                sigmask = 0 0 0 0
21268/1:              siginfo: SIG#0
21268/1:        lwp_sigtimedwait(0xFFBFF5AC, 0xFFBFF3E0, 0x00000010) = 0
21268/1:                sigmask = 0x00002000 0 0 0
21268/1:              siginfo: SIG#8192
21268/1:        lwp_sigtimedwait(0xFFBFF3D0, 0xFFBFF458, 0x00000010) = 0
21268/1:                sigmask = 0x00002000 0 0 0
21268/1:              siginfo: SIG#8192
21268/1:        sigprocmask(3, 0xFFBFF458, 0x00000000)          = 0
21268/1:                 set = 0x00002000 0 0 0
21268/1:        lwp_sigtimedwait(0xFF1D7C18, 0xFFBFF1B0, 0x00000010) = 0
21268/1:                sigmask = 0xFFBFFEFF 0x00001FFF 0 0
21268/1:              siginfo: SIG#-4194561
21268/1:        lwp_sigtimedwait(0xFFBFF1A0, 0xFFBFF228, 0x00000010) = 0
21268/1:                sigmask = 0xFFBFFEFF 0x0000FF1F 0 0
21268/1:              siginfo: SIG#-4194561
21268/1:        sigprocmask(3, 0xFFBFF228, 0xFFBFF238)          = 0
21268/1:                 set = 0xFFBFFEFF 0x0000FF1F 0 0
21268/1:                oset = 0x00002000 0 0 0
21268/1:        lwp_sigtimedwait(0xFFBFF238, 0xFFBFF1B0, 0x00000010) = 0
21268/1:                sigmask = 0x00002000 0 0 0
21268/1:              siginfo: SIG#8192
21268/1:        lwp_sigtimedwait(0xFFBFF1A0, 0xFFBFF360, 0x00000010) = 0
21268/1:                sigmask = 0x00002000 0 0 0
21268/1:              siginfo: SIG#8192
21268/1:        lwp_park(1, 1, 1)                               = 0
21268/1:        lwp_sigtimedwait(0xFFBFF370, 0xFFBFF088, 0x000001C0) = 0
21268/1:                sigmask = 0x0000082F 0 0 0
21268/1:              siginfo: SIG#2095
21268/1:        lwp_sigtimedwait(0xFFBFF090, 0xFFBFEFF8, 0x00000010) = 0
21268/1:                sigmask = 0 0 0 0
21268/1:              siginfo: SIG#0
21268/1:        lwp_sigtimedwait(0xFFBFEFE8, 0xFFBFF090, 0x00000010) = 0
21268/1:                sigmask = 0 0 0 0
21268/1:              siginfo: SIG#0
21268/1:        context(1, 0xFFBFF088)
21268/1:        getpid()                                        = 21268 [1417]
21268/1:        kill(1417, 0x0000000F)                          = 0
21268/1:        alarm(1)                                        = 0
21268/1:            Received signal #14, SIGALRM, in pause() [caught]
21268/1:              siginfo: SIG#0
21268/1:        pause()                                         Err#4 EINTR

No related posts.

Sun’s Java Web Console is a nice little toy. Great for those people who prefer the less extreme learning curve of a CLI, and great for manager types who like pictures.

I’ve used it several times before, but thought I’d try it in on x86 Solaris10u6 box I have here. Bloody Sun. You think a released version implies that it’s stable, and working.

For me the issue was that when I originally installed the server, I used a temporary hostname, and DHCP for network because I couldn’t be bothered to change my local DNS and figure out a spare IP. I then changed to fixed IP and hostname later.

The ramifications of this is that my hostname ended up being assigned to localhost, (aka 127.0.0.1). This means that when the WebConsole was initially installed and configured it used 127.0.0.1 throughout it’s hardcoded, (stupid Tomcat — hate it with a vengence), config files.

This is something that annoys the heck out of me: I understand developers making ‘packages’ that they can ‘deploy’ to a web server, (Tomcat). It’s nice to be able to ‘deploy’ a package. But for heck’s sake, why shove hostname/IP/port definitions in not just one file, but 50 stupid files that are zipped up along with shared libs and binaries? Why? Haven’t they heard of /etc/hosts, or even putting these definitions into somewhere common? Why do they have to have these definitions in multiple places?

Another great example of letting the developer take the lead in systems management.

Anyway, digging around the net trying to resolve my issue showed up a lot of similar complaints, but nothing really to sort it out. ‘wcadmin’ refused to start the ‘deployed’ packages, because they couldn’t connect to the Tomcat server running on my old hostname.

root@somewhere % wcadmin list

Deployed web applications (application name, context name, status):

console  ROOT            [stopped]
console  com_sun_web_ui  [stopped]
console  console         [stopped]
console  manager         [stopped]
console  zfs             [stopped]

Registered jar files (application name, identifier, path):

console  audit_jar     /usr/lib/audit/Audit.jar
console  console_jars  /usr/share/webconsole/lib/*.jar
console  jato_jar      /usr/share/lib/jato/jato.jar
console  javahelp_jar  /usr/jdk/packages/javax.help-2.0/lib/*.jar
console  shared_jars   /usr/share/webconsole/private/container/shared/lib/*.jar

Registered login modules (application name, service name, identifier):

console  ConsoleLogin  userlogin
console  ConsoleLogin  rolelogin

Persistent Jvm options:

-server
-Xmx128m
-XX:+UseParallelGC
-XX:ParallelGCThreads=4

I tried using wcadmin ro ‘undeploy’ and ‘deploy’ packages, and even the brut force method of

root@somewhere % find /var/webconsole /usr/share/webconsole /etc/webconsole /usr/lib/webconsole -type f | xargs egrep -l '127.0.0.1|localhost'

and manually modifying any config files found.

In the end I did a pkgrm/remove configs/pkgadd cycle:

1. Remove WebConsole packages.

system      SUNWasac  Sun Java System Application Server Admin Client
application SUNWmcon  Sun Java(TM) Web Console 3.0.2 (Core)
system      SUNWmconr Sun Java(TM) Web Console 3.0.2 (Root)
application SUNWmcos  Implementation of Sun Java(TM) Web Console (3.0.2) services
application SUNWmcosx Implementation of Sun Java(TM) Web Console (3.0.2) services
application SUNWmctag Sun Java(TM) Web Console 3.0.2 (Tags & Components)
application SUNWzfsgr ZFS Administration for Sun Java(TM) Web Console (Root)
application SUNWzfsgu ZFS Administration for Sun Java(TM) Web Console (usr)

The usual removal:

root@somewhere % pkgrm SUNWzfsgu SUNWmcosx SUNWmcos SUNWasac SUNWmcon SUNWzfsgr SUNWmconr SUNWmctag

Then remove all the config files:

root@somwehere % rm -rf /var/webconsole /usr/share/webconsole /etc/webconsole /usr/lib/webconsole

Or tarball up and then remove:

root@somewhere % tar cvf /tmp/webconsole.tar /var/webconsole /usr/share/webconsole /etc/webconsole /usr/lib/webconsole
root@somewhere % rm -rf /var/webconsole /usr/share/webconsole /etc/webconsole /usr/lib/webconsole

2. Re-install the WebConsole packages:
If you have your original install DVD/CD handy, then just chuck it in, and install the packages from the Product directory:

root@somewhere % pkgadd -d /cdrom/cdrom0/s2/Solaris_10/Product SUNWzfsgu SUNWmcosx SUNWmcos SUNWasac SUNWmcon SUNWzfsgr SUNWmconr SUNWmctag
The following packages are available:
 1  SUNWasac      Sun Java System Application Server Admin Client
 (i386) 8.2,REV=2006.08.16.08.46
 2  SUNWmcon      Sun Java(TM) Web Console 3.0.2 (Core)
 (i386) 3.0.2,REV=2006.12.08.20.48
 3  SUNWmconr     Sun Java(TM) Web Console 3.0.2 (Root)
 (i386) 3.0.2,REV=2006.12.08.23.39
 4  SUNWmcos      Implementation of Sun Java(TM) Web Console (3.0.2) services
 (i386) 3.0.2,REV=2006.12.08.23.39
 5  SUNWmcosx     Implementation of Sun Java(TM) Web Console (3.0.2) services
 (i386) 3.0.2,REV=2006.12.08.23.39
 6  SUNWmctag     Sun Java(TM) Web Console 3.0.2 (Tags & Components)
 (i386) 3.0.2,REV=2006.12.08.20.48
 7  SUNWzfsgr     ZFS Administration for Sun Java(TM) Web Console (Root)
 (i386) 1.0,REV=2006.10.24.20.44
 8  SUNWzfsgu     ZFS Administration for Sun Java(TM) Web Console (usr)
 (i386) 1.0,REV=2006.10.24.20.44

Even though you are installing version 3.1 of the Java Web Console, it actually appears as 3.0.2 for some stupid reason.

Everything will startup as expected using the new hostname defined.

root@somewhere % wcadmin list

Deployed web applications (application name, context name, status):

console  ROOT            [running]
console  com_sun_web_ui  [running]
console  console         [running]
console  manager         [running]
console  zfs             [running]

Registered jar files (application name, identifier, path):

console  audit_jar     /usr/lib/audit/Audit.jar
console  console_jars  /usr/share/webconsole/lib/*.jar
console  jato_jar      /usr/share/lib/jato/jato.jar
console  javahelp_jar  /usr/jdk/packages/javax.help-2.0/lib/*.jar
console  shared_jars   /usr/share/webconsole/private/container/shared/lib/*.jar

Registered login modules (application name, service name, identifier):

console  ConsoleLogin  userlogin
console  ConsoleLogin  rolelogin

Persistent Jvm options:

-server
-Xmx128m
-XX:+UseParallelGC
-XX:ParallelGCThreads=4

No related posts.

% pkg install gnome

Or

% pkg remove hugin

Or

% pkg verify firefox

Or

% pkg list

I mean.… Why not Sun?

(And what about my cat?)

No related posts.

The ‘ERROR: (_app_set_slice) Disk is not selected’ error message.

So, you’re trying to setup some DiskSuite metadevices during the jumpstart installation. You are getting this error because you don’t have the slices properly defined in your JET template. You will need to have something similar to the following defined in your JET template file, (under /opt/SUNWjet/Templates):

base_config_profile_root=512
base_config_profile_swap=16384
base_config_profile_s3_mtpt="/usr"
base_config_profile_s3_size="8192"
base_config_profile_s4_mtpt="/opt"
base_config_profile_s4_size="2048"
base_config_profile_s5_mtpt="/var"
base_config_profile_s5_size="2048"
base_config_profile_s6_mtpt=""
base_config_profile_s6_size=""

base_config_profile_disk_c0t1d0s0_mtpt="free"
base_config_profile_disk_c0t1d0s0_size="512"
base_config_profile_disk_c0t1d0s1_mtpt="free"
base_config_profile_disk_c0t1d0s2_size="8192"
base_config_profile_disk_c0t1d0s3_mtpt="free"
base_config_profile_disk_c0t1d0s3_size="8192"
base_config_profile_disk_c0t1d0s4_mtpt="free"
base_config_profile_disk_c0t1d0s4_size="2048"
base_config_profile_disk_c0t1d0s5_mtpt="free"
base_config_profile_disk_c0t1d0s5_size="2048"
base_config_profile_disk_c0t1d0s6_mtpt=""
base_config_profile_disk_c0t1d0s6_size=""
sds_root_mirror="c0t1d0"
sds_database_locations="c0t0d0s7:3 c0t1d0s7:3"
sds_database_partition="c0t0d0s7:64 c0t1d0s7:64"
sds_device_numbers="/:d100:d0:d10 swap:d101:d1:d11 /usr:d103:d3:d13 /opt:d104:d4:d14 /var:d105:d5:d15"

I had this error when I hadn’t specified the *_mtpt and set it to’free’, I had just left it blank.

Jumpstart directory not found

I had the following issue and it took me a while to figure it out.

Searching for JumpStart directory...
not found
Warning: Could not find matching rule in rules.ok
Press the return key for an interactive Solaris install program...

In the end I started to debug the ‘install-solaris’, and ‘profind’ scripts line by line. It turns out that I had defined my hosts with FQDN in my /etc/hosts file. When the server starts to look for the jumpstart directory it’ll use the /usr/sbin/install.d/profind s
cript to determine this. Contained within that is the line:

 set -- `/sbin/bpgetfile -retries 1 install_config`

Because I was using the standard RARP method of booting, bpgetfile was called to determine where to nfs mount the jumpstart directory. bootparams didn’t return any information from the boot server, because it was asking for ‘hostname’, rather than ‘hostname.domain.name’. So, if you get this issue. run the command

/sbin/bpgetfile -retries 1 install_config

from your install shell.

If you get nothing, then this is your issue. You will need to ensure you define your hosts not with a FQDN.

Actually, in my travels on this issue. Most of the resolutions to ‘no matching rule found in rules.ok’ issues had to do with accessing the jumpstart server from the client. So, check all your network settings, (netmasks, routes, etc). If not, then the above resolution will be it. On my clients, this was all OK, so it was baffling for a while.

Also, make sure that you have the matching swap and root partitions set correctly:

base_config_profile_root=512
base_config_profile_disk_c0t1d0s0_mtpt="free"
base_config_profile_disk_c0t1d0s0_size="512"
base_config_profile_swap=8192
base_config_profile_disk_c0t1d0s1_mtpt="free"
base_config_profile_disk_c0t1d0s1_size="8192"

No related posts.

Pre-requisites

• You need a stock Solaris 10 install.

• You need to have a Clearcase release repository with at least patch 2003.06.00–29 applied.

Clearcase zone

If you don’t know how to setup a Solaris 10 zone, then you can start off with a basic one by doing the following. Take particular note of the LOFS mounting of /kernel. That’s important.

% zonecfg -z zone-clearcase
zone-clearcase: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zone-clearcase> create
zonecfg:zone-clearcase> set autoboot=true
zonecfg:zone-clearcase> set zonepath=/mnt/zone/zone-clearcase
zonecfg:zone-clearcase> add net
zonecfg:zone-clearcase:net> set address=10.0.0.5
zonecfg:zone-clearcase:net> set physical=hme0
zonecfg:zone-clearcase:net> end
zonecfg:zone-clearcase> add fs
zonecfg:zone-clearcase:fs> set dir=/kernel
zonecfg:zone-clearcase:fs> set special=/kernel
zonecfg:zone-clearcase:fs> set type=lofs
zonecfg:zone-clearcase:fs> set options ro,nosuid
zonecfg:zone-clearcase:fs> end
zonecfg:zone-clearcase> info
zonepath: /mnt/zones/zone-clearcase
autoboot: true
pool:
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
fs:
        dir: /kernel
        special: /kernel
        raw not specified
        type: lofs
        options: [ro,nosuid]
net:
        address: 10.0.0.5
        physical: hme0
zonecfg:zone-clearcase> verify
zonecfg:zone-clearcase> commit
zonecfg:zone-clearcase> exit
% 

Now install the zone, that has just been configured.

% mkdir -p /mnt/zones/zone-clearcase
% chmod 700 /mnt/zones/zone-clearcase
% zoneadm -z zone-clearcase install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <143885> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <1025> packages on the zone.
Initialized <1025> packages on zone.
Zone  is initialized.
Installation of <1> packages was skipped.
Installation of these packages generated warnings: 
The file  contains a log of the zone installation.

Now you can boot up the zone, login to it’s console, and finish off the install.

% zoneadm list -cv     % zoneadm list -cv
    ID NAME             STATUS         PATH
     0 global           running        /
     - zone-clearcase   installed      /mnt/zones/zone-clearcase
  % zoneadm -z zone-clearcase boot
  % zlogin -C zone-clearcase
  [Connected to zone 'zone-clearcase' console]
  100/100
  What type of terminal are you using?
   1) ANSI Standard CRT
   2) DEC VT52
   3) DEC VT100

etc etc etc

Now you need to setup the clearcase repository. I gather you have done this before, so I won’t spell it out here. Everything is standard issue — install what you want leave out the bits you don’t want.

Once you have your clearcase repository you will need to install clearcase first in the global zone. Don’t worry — you will be disabling this later on. I also gather that you have performed this before, so I’ll leave that out as well.

Now the fun begins. Shutdown clearcase in the global zone.

% /etc/init.d/clearcase stop

Edit the /etc/init.d/clearcase and put the following entries right at the start.

% /usr/sbin/modload /opt/rational/clearcase/sun5/kvm/5.10/sparcv9/mvfs
% exit    

Now startup ‘clearcase’ in the global zone. All this will do is load up the kernel module ‘mvfs’

Next you will have to restart your clearcase zone. This will ensure that this zone will be able to see the mvfs module loaded up.

% zoneadm -z zone-clearcase reboot

Now you can install clearcase into the zone you have created above. Do the normal things, install what you want leave out the bits you don’t. However, right at the end of the install you will possibly get some errors. You can ignore these.

And that’s it! You have one working clearcase server in a zone. You can setup as many as you want as well.

No related posts.