kkjcre1p: unable to spawn jobq slave process
Process J000 died, kkjcre1p: unable to spawn jobq slave process
================================================================
PROCESS J000 And M000 Die,
this might be related to a lack of OS-resources.
Using OSWatcher is a good tool to get more evidence of this.
or
use top/vmstat/prstat to check it.
Normally OSWatcher is taking snapshots every 30 seconds and in the above output it shows that several snapshots are missing. OSWatcher is using OS-commands like 'top' and 'vmstat'. So if they are having problems, than there is a serious problem on OS-level.
Another indication of the OS problems are the very high number of processes in the CPU Run and Blocked Q.
Alert log shows like belows-
------------------------------------------------
Process PN82 died
Tue Feb 22 16:42:29 2011
Process J000 died
Tue Feb 22 16:42:29 2011
kkjcre1p: unable to spawn jobq slave process
Tue Feb 22 16:42:29 2011
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1880.trc:
Tue Feb 22 16:42:30 2011
Process PN83 died
Tue Feb 22 16:42:32 2011
Process PN84 died
Tue Feb 22 16:42:33 2011
Process PN85 died
Tue Feb 22 16:42:34 2011
Process PN86 died
Tue Feb 22 16:42:35 2011
Process PN87 died
Tue Feb 22 16:42:36 2011
Process J000 died
Tue Feb 22 16:42:36 2011
kkjcre1p: unable to spawn jobq slave process
Tue Feb 22 16:42:36 2011
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1880.trc:
Tue Feb 22 16:42:37 2011
Process PN88 died
Tue Feb 22 16:42:40 2011
-----------------------------------------------
=======================
or
===================
"unable to spawn jobq slave process" this message can be show for
not correct setting of JOB_QUEUE_PROCESSES .
How to determine the correct setting for JOB_QUEUE_PROCESSES
============================================================
Solution
==============
The ideal setting for JOB_QUEUE_PROCESSES should be set to the maximum number of jobs that would ever be run concurrently on a system PLUS a few more.
1) Initially set JOB_QUEUE_PROCESSES higher than the most number of jobs that could ever run concurrently
Be careful not to set this too high as it can effect database performance or even exhaust OS resources
A fair estimate can be done by looking at DBA_JOBS ... NEXT_DATE and NEXT_SEC columns
The reason that this is a fair estimate is the JOB SCHEDULER submits one time execute jobs to the queue
for execution when it chooses to run a job ... so it is possible that the above estimate would be a good bit
off ... Example ... Running or UTLRP.sql ... on a system with multiple CPU's ... will cause he SCHEDULER
to submit multiple jobs ... at the same time ... to the job queue
NOTE: Add one more to your estimate for the job that will be created below
2) Use the job queue to monitor itself
a) Login as SYS
b) Create the table to do the monitoring
CREATE TABLE MAX_CONCURRENT_JOBS (
SAMPLE_DATE DATE,
JOBS_RUNNING NUMBER);
c) Create the job to do the monitoring
CREATE OR REPLACE PROCEDURE MAX_CONCURRENT_JOBS_MONITOR IS
BEGIN
INSERT INTO MAX_CONCURRENT_JOBS
SELECT SYSDATE, COUNT(*) FROM DBA_JOBS_RUNNING;
COMMIT;
END;
/
d) Create the job queue job to do the monitoring
DECLARE
v_jobno number;
BEGIN
DBMS_JOB.SUBMIT(
JOB => v_jobno,
WHAT => 'MAX_CONCURRENT_JOBS_MONITOR;',
NEXT_DATE => SYSDATE,
INTERVAL => 'SYSDATE + (5/1440)');
COMMIT;
END;
/
NOTE : The example above uses a 5 minute interval (5/1440) ... this can be adjusted as needed
e) Monitor MAX_CONCURRENT_JOBS
SELECT MAX(JOBS_RUNNING) FROM MAX_CONCURRENT_JOBS;
The length of time for monitoring depends on the INTERVAL in the job queue for ALL jobs ... It is recommended that at least two (2) executions of every job in the job queue be monitored with this process
3) Set JOB_QUEUE_PROCESSES to the appropriate setting
ALTER SYSTEM SET JOB_QUEUE_PROCESSES =SCOPE = ...
4) Remove the monitoring system
SELECT JOB FROM DBA_JOBS WHERE WHAT='MAX_CONCURRENT_JOBS_MONITOR;';
EXEC DBMS_JOB.REMOVE();
DROP PROCEDURE MAX_CONCURRENT_JOBS_MONITOR;
DROP TABLE MAX_CONCURRENT_JOBS;
NOTE : this process needs to be done periodically ... especially if new jobs are being added to the job queue
and also see here:
http://halimdba.blogspot.com/2010/12/solaris-error-32-broken-pipe-or-linux.html
================================================================
PROCESS J000 And M000 Die,
this might be related to a lack of OS-resources.
Using OSWatcher is a good tool to get more evidence of this.
or
use top/vmstat/prstat to check it.
Normally OSWatcher is taking snapshots every 30 seconds and in the above output it shows that several snapshots are missing. OSWatcher is using OS-commands like 'top' and 'vmstat'. So if they are having problems, than there is a serious problem on OS-level.
Another indication of the OS problems are the very high number of processes in the CPU Run and Blocked Q.
Alert log shows like belows-
------------------------------------------------
Process PN82 died
Tue Feb 22 16:42:29 2011
Process J000 died
Tue Feb 22 16:42:29 2011
kkjcre1p: unable to spawn jobq slave process
Tue Feb 22 16:42:29 2011
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1880.trc:
Tue Feb 22 16:42:30 2011
Process PN83 died
Tue Feb 22 16:42:32 2011
Process PN84 died
Tue Feb 22 16:42:33 2011
Process PN85 died
Tue Feb 22 16:42:34 2011
Process PN86 died
Tue Feb 22 16:42:35 2011
Process PN87 died
Tue Feb 22 16:42:36 2011
Process J000 died
Tue Feb 22 16:42:36 2011
kkjcre1p: unable to spawn jobq slave process
Tue Feb 22 16:42:36 2011
Errors in file /d04/admin/stlbas/bdump/stlbas_cjq0_1880.trc:
Tue Feb 22 16:42:37 2011
Process PN88 died
Tue Feb 22 16:42:40 2011
-----------------------------------------------
=======================
or
===================
"unable to spawn jobq slave process" this message can be show for
not correct setting of JOB_QUEUE_PROCESSES .
How to determine the correct setting for JOB_QUEUE_PROCESSES
============================================================
Solution
==============
The ideal setting for JOB_QUEUE_PROCESSES should be set to the maximum number of jobs that would ever be run concurrently on a system PLUS a few more.
1) Initially set JOB_QUEUE_PROCESSES higher than the most number of jobs that could ever run concurrently
Be careful not to set this too high as it can effect database performance or even exhaust OS resources
A fair estimate can be done by looking at DBA_JOBS ... NEXT_DATE and NEXT_SEC columns
The reason that this is a fair estimate is the JOB SCHEDULER submits one time execute jobs to the queue
for execution when it chooses to run a job ... so it is possible that the above estimate would be a good bit
off ... Example ... Running or UTLRP.sql ... on a system with multiple CPU's ... will cause he SCHEDULER
to submit multiple jobs ... at the same time ... to the job queue
NOTE: Add one more to your estimate for the job that will be created below
2) Use the job queue to monitor itself
a) Login as SYS
b) Create the table to do the monitoring
CREATE TABLE MAX_CONCURRENT_JOBS (
SAMPLE_DATE DATE,
JOBS_RUNNING NUMBER);
c) Create the job to do the monitoring
CREATE OR REPLACE PROCEDURE MAX_CONCURRENT_JOBS_MONITOR IS
BEGIN
INSERT INTO MAX_CONCURRENT_JOBS
SELECT SYSDATE, COUNT(*) FROM DBA_JOBS_RUNNING;
COMMIT;
END;
/
d) Create the job queue job to do the monitoring
DECLARE
v_jobno number;
BEGIN
DBMS_JOB.SUBMIT(
JOB => v_jobno,
WHAT => 'MAX_CONCURRENT_JOBS_MONITOR;',
NEXT_DATE => SYSDATE,
INTERVAL => 'SYSDATE + (5/1440)');
COMMIT;
END;
/
NOTE : The example above uses a 5 minute interval (5/1440) ... this can be adjusted as needed
e) Monitor MAX_CONCURRENT_JOBS
SELECT MAX(JOBS_RUNNING) FROM MAX_CONCURRENT_JOBS;
The length of time for monitoring depends on the INTERVAL in the job queue for ALL jobs ... It is recommended that at least two (2) executions of every job in the job queue be monitored with this process
3) Set JOB_QUEUE_PROCESSES to the appropriate setting
ALTER SYSTEM SET JOB_QUEUE_PROCESSES =
4) Remove the monitoring system
SELECT JOB FROM DBA_JOBS WHERE WHAT='MAX_CONCURRENT_JOBS_MONITOR;';
EXEC DBMS_JOB.REMOVE(
DROP PROCEDURE MAX_CONCURRENT_JOBS_MONITOR;
DROP TABLE MAX_CONCURRENT_JOBS;
NOTE : this process needs to be done periodically ... especially if new jobs are being added to the job queue
and also see here:
http://halimdba.blogspot.com/2010/12/solaris-error-32-broken-pipe-or-linux.html
