Last week we upgraded our Master and Media servers to Netbackup 7.6.0.4 from 7.5.0.7 completing on the 17th April. Everything worked fine until 19th April when the we noticed Oracle and SQL backups were failing with an error 6 and an Exchange backup which normally takes 2.5 hours was still running after 2 days. Client servers are running a variety of versions from 7.1 to 7.5.0.7. We have two media servers and we only have this problem on one media server and the problem only affects some of the servers.

Error codes we are getting are 6, 41 and 196.

Investigating the problem the we found that the problem is the backups to the MSDP storage times out or is very slow in the case of the Exchange backup. Looking in the logs I have seen the following errors:

bptm

00:00:56.003 [7068.8068] <16> 86851:bptm:7068:hpwp-med01-bkp.hcuk.pri: [ERROR] PDVFS: pdvfs_lib_log: mtstrm_is_server_started: Cr_ErrnoException: Get Process Status command failed: Cr_ErrnoException: Timed out after waiting 1200s to send command Get Process Status to mtstrmd
00:00:56.003 [7068.8068] <16> 86851:bptm:7068:hpwp-med01-bkp.hcuk.pri: [ERROR] PDVFS: PdvfsMtstrmStartDaemon: The multi-threaded streaming daemon is not ready to receive requests even after 4 attempts.  Check if mtstrmd is running and look for errors in mtstrmd.log.
00:00:56.003 [7068.8068] <16> 86851:bptm:7068:hpwp-med01-bkp.hcuk.pri: [ERROR] PDSTS: start_mtstrm_session: PdvfsMtstrmStartDaemon failed (18:Improper link)

00:30:28.676 [8116.13800] <16> 84077:bptm:8116:hpwp-med01-bkp.hcuk.pri: libsts openp() 15/04/21 00:30:28: stsm_open_module C:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspizapi.dll failed err 2060053 platerr 193: LoadLibrary

dbclient

00:45:29.702 [11258] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/11258.0.1429572612

01:00:30.755 [11258] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/11258.0.1429572612

01:00:30.756 [11258] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/11258.0.1429572612>

01:00:30.756 [11258] <16> CreateNewImage: ERR - serverResponse() failed

01:00:30.756 [11258] <16> VxBSACreateObject: ERR - Could not create new image with file /arch_ultq4t6kk_s17085_p1_t877566612.

bpdm

00:00:17.860 [900.10140] <16> 87153:bptm:900:hpwp-med01-bkp.hcuk.pri: libsts openp() 15/04/22 00:00:17: stsm_open_module C:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspizapi.dll failed err 2060053 platerr 193: LoadLibrary failed

These errors seem to show the backups are unable to write to the storage as the backups seem to go to the media server ok and then fail because they cannot write to the storage. Apart from the upgrade the Netbackup configuration has not changed.

My gut feeling is that the failures are not caused by the upgrade and possibly not by Netbackup as these errors seem to be a consquence rather than a cause and we had at least a days worth of successful backups beform we started to have problems but as always for a Netbackup Administrator the problem is proving it.

Is anyone able give me any ideas? I've worked on this for 2 days and I am out of them.

Hello everyone

Somebody can help me to check where is the reason about the status code. Last week we had to  insert on the library 100 cartridges to minimize this kind of errors, but today I see the same error. Could be I don't have the enough STU, but I need to know if is this the reason. Which is the log to check this?

May be I have to organize my SLP's, someby have any idea about how can I do it?

As can you see, this are the messages about it:

22-abr-2015 5:00:59 - requesting resource LCM_media_server01-hcart-robot-tld-0
22-abr-2015 5:00:59 - granted resource  LCM_media_server01-hcart-robot-tld-0
22-abr-2015 5:01:00 - begin Duplicate
22-abr-2015 5:01:00 - started process RUNCMD (pid=27132102)
22-abr-2015 5:01:01 - ended process 0 (pid=27132102)
22-abr-2015 5:01:01 - requesting resource media_server01-hcart-robot-tld-0
22-abr-2015 5:01:01 - requesting resource @aaaaj
22-abr-2015 5:01:01 - reserving resource @aaaaj
22-abr-2015 5:01:02 - end Duplicate; elapsed time 0:00:02
22-abr-2015 5:01:02 - Error nbjm (pid=25231436) NBU status: 96, EMM status: No media is available
22-abr-2015 5:01:02 - Error nbjm (pid=25231436) NBU status: 96, EMM status: No media is available
22-abr-2015 5:01:02 - Error nbjm (pid=25231436) NBU status: 96, EMM status: No media is available
unable to allocate new media for backup, storage unit has none available  (96)

How would I configure the backup_exit_notify script to email me when certain clients have failed their backups and be able to filter the emails with respect to the policy names?

Hi All;

Can someone tell me :" how many devices we can have on one media server " Thank you.

Hello Experts,

We have 2 media servers and 1 master server. storage units are on 1 master and 2 media servers.

Few days back we had a strange issue tapes are taking too much time to mount on some drives.

We have SSO 

Master Server:- Win2008 R2

Media Servers:- Solaris 11 

We were given many solutions from Symantec TSE one of these is as below

The system is split between scan hosts.
Recommend adjusting the scan host to just one server. (Select a server with abundant resources).

To reduce the chances of the master server becoming the scan host set scanability 1.
Likewise to increase the changes of a media server becoming the only use scanability 9.
 

So question is as we have many backups running on master servers storage unit... Is there any impact if we change the scan host scanbility to 1 on master?

Good morning,

I'm trying to delete a storage lifecycle policy but i'm getting the below message.

The storage lifecycle policy, could not be deleted. Status: 1519

Images are in process. Please, run the nbstlutil command to get a list of those images if you need to cancel or expire them.

How do i expire the images?

Thanks,

i need the information from the Host Propierties (client) to perform a migration plan to move to a newest version of nbu Client , its possible to export that information to an excell File ?

thanks a lot .

We're using Netbackup 7.1.0.3 (upgrading is not an option) and have the following configuration set using Blat to send email:

Master Server:
Global Attribute: backupfailures@domain.com
Universal Settings: Administrator: Server Sends Mail (no recipient)

Majority of clients:
Universal Settings: Administrator: Server Sends Mail (no recipient)
 

We then have a small handful of clients configured as follows:
Universal Settings: Administrator: Server Sends Mail (auditlogs@domain.com)

The intent is that all client backups generate an alert to backupfailures@domain.com if a backup fails, no alert if a backup is successful.

A small handful of clients besides generating the failed backup alerts ONLY to backupfailures@domain.com, we ALSO want to send a failure AND success messages to auditlogs@domain.com, those clients have the client universal setting configured with the auditlogs recipient.

For some reason, one of the servers which we want to send both failed backup alerts to "backupfailures" and BOTH failed and successful alerts to "auditlogs", is not sending alerts on failed backups to "backupfailures" as configured in the master server's universal settings. It IS sending both successful and failed alerts to "auditlogs" as configured in the client setting.

I have created a new policy to test with and noticed that if I perform a Windows backup of a file, the alerts work as desired but if I change it to a SQL based backup, the alerts do NOT work as desired.

We're using blat, and I placed a line at the end of nbmail.cmd to echo out the values of %1, %2, %3, and %4 to a log to see exactly what Netbackup is sending to blat and noticed that when SQL backups on this server fail, it does NOT even call blat. I have been working with Symantec support to troubleshoot this, but am not getting anywhere. I'm wondering if anyone else has ever noticed similar behavior or has any ideas?

Hello all,

I'm trying to make BRM on Oracle VM LDOM for Solaris 11.

I have the following error saying that there is no /cdrom.

cdrom on ldom is mount as disk on /.cdrom

bellow the message.

can someone help me resolv that issue?

Enter the NetBackup client name : client
Enter the client's netmask (dotted decimal form) : 192.168.200.8

Enter the client's netmask (dotted decimal form) : 255.255.255.0
Enter the NetBackup master server name : master
Enter the NetBackup master server IP Address (dotted decimal form) : 192.168.200.1
Enter the client's default gateway (dotted decimal form) : 192.168.200.250
add net default: gateway 192.168.2.250
add persistent net default: gateway 192.168.2.250
Hostname: client
Apr 21 17:42:33 netbackup[1353]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:33 netbackup[1353]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1420]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1420]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1423]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1423]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1426]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1426]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1429]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:35 netbackup[1429]: OID:128:V-1-6 Unable to load configuration settings.
Reconfiguring /dev and /devices for Bare Metal Restore
Apr 21 17:42:37 netbackup[1439]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:37 netbackup[1439]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:37 netbackup[1442]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:37 netbackup[1442]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:38 netbackup[1446]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:38 netbackup[1446]: OID:128:V-1-6 Unable to load configuration settings.
ERROR: Shared Resource Tree /cdrom/ not mounted.
ERROR: Please boot with correct Media Boot CD or network boot image.
Apr 21 17:42:38 netbackup[1450]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:38 netbackup[1450]: OID:128:V-1-6 Unable to load configuration settings.
Apr 21 17:42:38 netbackup[1456]: OID:122:V-1-6 Unable to load configuration settings.
Apr 21 17:42:38 netbackup[1456]: OID:128:V-1-6 Unable to load configuration settings.

Bare Metal Restore has failed.
Now, you will be dropped to a shell prompt.
You may reboot the system when you have completed diagnosis.

 df -h
Filesystem             Size   Used  Available Capacity  Mounted on
/devices/ramdisk-root:a
                       233M   142M        91M    62%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   5.3G   2.0M       5.3G     1%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
/devices/virtual-devices@100/channel-devices@200/disk@0:a
                       630M   630M         0K   100%    /.cdrom
/dev/lofi/1            1.1G   1.1G         0K   100%    /usr
/dev/lofi/2            105M   105M         0K   100%    /mnt/misc
/mnt/misc/opt          105M   105M         0K   100%    /mnt/misc/opt
swap                   5.3G    16K       5.3G     1%    /root
swap                   5.3G    64K       5.3G     1%    /jack
swap                   5.3G   280K       5.3G     1%    /tmp

I had one master server and three media server

I had a problem in one of the media server where it wasn't able to control the robot so I deleted all devices from this media server  and even from all media servers and the masterserver itself using nbemmcmd -deletealldevices 

Mastersever name : Netbackup

Media Server : expowsrv

/usr/openv/netbackup/bin/admincmd/nbemmcmd -deletealldevices -machinename expowsrv -machinetype media
/usr/openv/netbackup/bin/admincmd/nbemmcmd -deletealldevices -machinename netbackup -machinetype media

The problem now is that the robot is seen from the media server expowsrv as local controlled by expowsrv

however from the netbackup it's seen but also controlled from the media server expowsrv 
this causes a problem in the robot inventory as I can't find the robot in Netbackup to do the inventory from it
so when I try to update the library contents using the robot controlled by expowsrv it gives me another error

it can only show the contents but failed to update
seems because all the previous inventories that were done during all these previous years was done by the robot that is locally attached and controlled by the master server 

It's worth mentioning that all

sgscan , scan can show the locally attached robot 

By the way both versions are 7.5 

Is there any ideas what's the problem ?

Hi,

I am currently trying to plan an upgrade of our whole Netbackup environment which consists of the following:

A master server and 2 Netbackup 5220 appliances at one site (site A).

A master server and 2 Netbackup 5220 appliances at another site (site B).

All versions of Netbackup are 7.6.0.1 (or 2.6.0.1 for the appliances).

Both sites conduct their own backups for the clients at that site and then replicate the images to the other site via AIR (site A replicates to site B and vice versa).

I wish to upgrade the whole environment to 7.6.1.1 and was wondering the best way to do it whilst minimising downtime etc.

For example, can I upgrade site A one week and then site B the next week or will the appliances being on different versions for a week affect replications?

Any advice would be greatly appreciated. Many Thanks,

Ade

Hi all,

can some one help me resolv that issue on MsSQL GUI?

I tryed to restore MsSQL Database en the GUI report the following : No backup SQL server object were found for client, in the specified timeframe

How can I resolve that issue

BR

Hi Experts,

We have one master server windows 2008 and 8 drives.

8 drives are visible on OS level and working fine.

We are experincing issue with the Drives 1 or 2 drives are taking too much time to mount tapes. Its an intermittent issue. Sometimes all 8 are working perfectly fine. We observed its not only a particular drive but if all 7 drives are working 1 of them is not mounting tape due to which backups are in queue and after half an hour it mounts the tape and all drives start working.

In scan -changer command on master server it shows all the 8 drives with serial number.

However scan command also shows all 8 drives with serial number and on specific each drives there are only 5 drives. Is there any issue?

or if scan -changer shows all the 8 drives is it ok? if its ok then why scan command is showing 5 drives( Indivudual details of the drives ) after showing all 8 drives ? 

Our backup server is NBU Enterprise Server, 7.6.1.1 running on OEL 5.10 (x86), on a Dell R620 server and Dell PVTL 2000 (2 x LTO5 drives), each drive connected via 6Gps SAS interface to NBU server. 

We have two Exchange 2010 servers, each DB over 420GB. I put in place a Windows 2008 R2 proxy server to handle the duplication of images from disk (BasicDisk) to tape. We need the GRT option as we need to recover items at a message level. 

What has been erratically happening is that duplication jobs fail with a 191 status and the dreaded:
04/23/2015 09:08:17 - Error bpduplicate (pid=828) db_IMAGE() failed: database system error (220)
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Status = no images were successfully processed.
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Duplicate of backupid exch2_1429750827 failed, database system error (220).
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Status = no images were successfully processed.
 

So... been going back and forth with support and for some reason, today I looked at the Application Log on the proxy server and found:

Error      4/23/2015 9:08:16 AM    Application Error              1005       (100)

Error      4/23/2015 9:08:16 AM    Application Error              1000       (100)

Error      4/23/2015 9:08:16 AM    Application Error              1000       (100)

The detail of the Event ID 1000 is:

Log Name:      Application
Source:        Application Error
Date:          4/23/2015 9:08:16 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      GRTPROXY
Description:
Faulting application name: nblbc.exe, version: 7.601.115.212, time stamp: 0x54dcea59
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000006
Fault offset: 0x0000000000029c42
Faulting process id: 0x5a0
Faulting application start time: 0x01d07dbf25e6bbf8
Faulting application path: C:\Program Files\Veritas\NetBackup\bin\nblbc.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: cd8dd4eb-e9b9-11e4-8564-0050568d7352
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-04-23T13:08:16.000000000Z" />
    <EventRecordID>4001</EventRecordID>
    <Channel>Application</Channel>
    <Computer>GRTPROXY</Computer>
    <Security />
  </System>
  <EventData>
    <Data>nblbc.exe</Data>
    <Data>7.601.115.212</Data>
    <Data>54dcea59</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7601.18247</Data>
    <Data>521eaf24</Data>
    <Data>c0000006</Data>
    <Data>0000000000029c42</Data>
    <Data>5a0</Data>
    <Data>01d07dbf25e6bbf8</Data>
    <Data>C:\Program Files\Veritas\NetBackup\bin\nblbc.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>cd8dd4eb-e9b9-11e4-8564-0050568d7352</Data>
  </EventData>
</Event>

And the detail of EventID 1005 is as follows:

Log Name:      Application
Source:        Application Error
Date:          4/23/2015 9:08:16 AM
Event ID:      1005
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      GRTPROXY
Description:
Windows cannot access the file  for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program NetBackup because of this error.

Program: NetBackup
File: 

The error value is listed in the Additional Data section.
User Action
1. Open the file again. This situation might be a temporary problem that corrects itself when the program runs again.
2. If the file still cannot be accessed and
    - It is on the network, your network administrator should verify that there is not a problem with the network and that the server can be contacted.
    - It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.
3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.
4. If the problem persists, restore the file from a backup copy.
5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for further assistance.

Additional Data
Error value: C00000C4
Disk type: 0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="49152">1005</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-04-23T13:08:16.000000000Z" />
    <EventRecordID>4003</EventRecordID>
    <Channel>Application</Channel>
    <Computer>GRTPROXY</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>NetBackup</Data>
    <Data>C00000C4</Data>
    <Data>0</Data>
  </EventData>
</Event>

Any ideas as to why nblbc would crash?

Hi,

I have some server didn't complete the backup and I receive ''socket write failed  (24)''.

If I look inside the log I have this information: ''ERR - Cannot write to STDOUT. Errno = 110: Connection timed out''

The connexion are good between client server and master. The full Backup are good but not the Differential Incremental Backup.

My client server is Linux Redhat 2.6.18 and my master is SunOS(5.10) version 7.5.0.6

What do you think ?

It's possible to put a trap for to see what happen during the backup ?

Thanks,

Hello everyone.

We have a Data Domain, model 4500 and we try to use DD Boost for Symantec NBU. But in this moment the configuration on the media servers has an issue. This is the problem:

Our  media server send the data over the LAN, this is because each server can't use the FC, we like to know if exist any restriction when NBU use a media server with OS on aix 7.1. The data in this moment is send over administration interface, 1 GBps, this is very slow.

We don't have 10 gigabit ethernet.

I really appreciate any information, recommentation or help to solve this issue.

I have a doubt, can I configure a SAN Client and send the data over FC? This client could be on any version of OS?

We use:

NBU 7.6.0.2 on the master and media server.

Management wants to know what exactly is being restored for each priority restore.  For instance, we restored a directory and want to find out all the files/folders under this directory that were restored.  Just want to ensure I have the correct way to get this info from the logs for future restores.

Is this info from the master logs or client?  I know on the master I have read on other posts to grab it from the bpbkar log directory (verbose L5).  Or on the client, create these 3 logs: bpinetd, bpcd, and tar, but this seems to capture connection logs.

Steps before restore:

Ensure the bpbkar directory is created

Turn logging up to L5

Begin restore

Turn logging down after restore completes

Let me know if I have something wrong.

Thanks

Hi All,

One of our customers wants to install another Netbackup on their environment to backup other servers. They already have an existing netbackup v7.6 but its located on other server farm. Its a capacity based licensing, so as far as i know, they still haven't reach their capacity

So they want to install a NBU Master on a Cent OS v6. I'm planning to install NBU 7.1 on the CentOS 6, since based on NBU SCL, only NBU 7.1 is available to install as a master server. Can I use their NBU v7.6 license on NBU v7.1? are there any conflicts?

Please provide me a direct link to where to download NBU 7.1

Thanks

Hi,

We are using  AIX 7.1 server and Netbackup 7.5. Only root user has access to Netbackup for backup and recovery. Root umask is set at 027, by right this would prevent any netbackup files generated to have the world writable format. However we found a few CatalogBackup.lck files which have a world writable file format in /usr/openv/netbackup/db/images/... directory. Could anyone help to suggest a permanent solution for this, since our audit requirement would not allow any world writable file.

Thank You.