Upgrading Dual Routing Engine Juniper MX Series
Table of Contents
In one of my previous post, I explained how you would go about upgrading a Juniper EX switch. I said whenever I got the chance to upgrade a MX Series Router, I’ll get something noted down…… raises hands today is the day! As I’ve said in a few posts, there has been a lot of change and now team is now getting access to the Core Juniper MX Series Routers. As part of this increased access, one of our first tasks is it upgrade JunOS from 12.3R5.7 to 14.1R6.4. With most, if not, all MX Series above the MX80, they will come with two Routing Engines (RE), and both are independent of each other. This is being the case, when upgrading a MX; you will need to upgrade each RE by individually.
This post will go over what you will need to do upgrade an MX Router, in my setup I’ll be upgrading a Juniper MX480 Router and I’ll be doing the upgrade via the console port on each Routing Engine.
To link two Routing Engines together, you would need to apply similar configuration to what I used:
set groups re0 system host-name re0-mx480
set groups re0 interfaces fxp0 unit 0 family inet address x.x.x.x/x
set groups re1 system host-name re1-mx480
set groups re1 interfaces fxp0 unit 0 family inet address x.x.x.x/x
set apply-groups re1
set apply-groups re0
set chassis redundancy graceful-switchover
set routing-options nonstop-routing
set system commit synchronize
With that all cleared up.. Let’s get cracking :)
Pre Works⌗
Upload the new firmware version to wherever you normally keep it them. Currently, we would normally upload the package into the /var/tmp folder on the device in question
[marquk01@netlab ~]$ scp jinstall-14.3R6.4-domestic-signed.tgz re0-mx480:/var/tmp
After just saying how you to link the two REs together, for an upgrade, you will need to disable graceful-switchover and nonstop-routing. Skipping this step can potentially result in the control plane and forwarding plane having two different JUNOS versions, which can cause a number of potential issues!
deactivate chassis redundancy graceful-switchover
deactivate routing-options nonstop-routing
Upgrade Process⌗
Having disabled both graceful-switchover and nonstop-routing, log onto the Backup RE, either by console or from the Master RE run the command request routing-engine login re1. Once on the Backup RE, you will need to run the command request system software validate add /var/tmp/xxx reboot.
marquk01@re1-mx480> request system software validate add /var/tmp/jinstall-14.1R6.4-domestic-signed.tgz reboot
If you’re like us and save the new firmware package to the local device, when you run the software add command DO NOT set what RE has the package stored. If you do you add the package’s location once the upgrade is completed, on one of the RE, it will delete the image from the device!
Additionally you had requested a session from re0 to re1 to connect to Backup RE, once the RE reboots, you get this message and get booted off
marquk01@re1-mx480>
*** FINAL System shutdown message from root@re1-mx480 ***
System going down IMMEDIATELY
rlogin: connection closed
If you have console access, you can watch the upgrade ticking along, if you don’t, you can confirm the Backup RE is up and running by using the command show chassis routing-engine, it will show the status and hardware stats for both Routing Engines.
show chassis routing-engine output⌗
marquk01@re0-mx480> show chassis routing-engine
Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)
Temperature 31 degrees C / 87 degrees F
CPU temperature 37 degrees C / 98 degrees F
DRAM 3584 MB (3584 MB installed)
Memory utilization 20 percent
CPU utilization:
User 0 percent
Background 0 percent
Kernel 4 percent
Interrupt 0 percent
Idle 96 percent
Model RE-S-2000
Serial ID 9012021718
Start time 2016-03-22 13:14:37 GMT
Uptime 3 hours, 38 minutes, 16 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
0.01 0.01 0.00
Routing Engine status:
Slot 1:
Current state Backup
Election priority Backup (default)
Temperature 33 degrees C / 91 degrees F
CPU temperature 38 degrees C / 100 degrees F
DRAM 3584 MB (4096 MB installed)
Memory utilization 16 percent
CPU utilization:
User 0 percent
Background 0 percent
Kernel 0 percent
Interrupt 0 percent
Idle 100 percent
Model RE-S-2000
Serial ID 9012022174
Start time 2016-03-22 16:47:56 GMT
Uptime 4 minutes, 45 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
0.34 0.47 0.23
Having upgraded Backup RE, to reduce the downtime and service impact, you can failover the Master Routing Engine, so that the Backup becomes the new Master Routing Engine. This is an manual process by running the command, from the current Master RE, request chassis routing-engine master switch. This WILL cause a brief outage as the PFE is reset and the new firmware is loaded.
marquk01@re1-mx480> request chassis routing-engine master switch
warning: Traffic will be interrupted while the PFE is re-initialized
Toggle mastership between routing engines ? \[yes,no\] (no) yes
Resolving mastership...
Complete. The other routing engine becomes the master.
You can confirm by running show chassis routing-engine again on RE0
AFTER failing over Routing Engine⌗
marquk01@re0-mx480> show chassis routing-engine
Routing Engine status:
Slot 0:
Current state Backup
Election priority Master (default)
Temperature 32 degrees C / 89 degrees F
CPU temperature 39 degrees C / 102 degrees F
DRAM 3584 MB (3584 MB installed)
Memory utilization 16 percent
CPU utilization:
User 2 percent
Background 0 percent
Kernel 1 percent
Interrupt 0 percent
Idle 97 percent
Model RE-S-2000
Serial ID 9012021718
Start time 2016-03-22 13:14:37 GMT
Uptime 3 hours, 50 minutes, 7 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
0.21 0.07 0.02
Routing Engine status:
Slot 1:
Current state Master
Election priority Backup (default)
Temperature 33 degrees C / 91 degrees F
CPU temperature 41 degrees C / 105 degrees F
DRAM 3584 MB (4096 MB installed)
Memory utilization 22 percent
CPU utilization:
User 43 percent
Background 0 percent
Kernel 28 percent
Interrupt 0 percent
Idle 29 percent
Model RE-S-2000
Serial ID 9012022174
Start time 2016-03-22 16:47:56 GMT
Uptime 16 minutes, 42 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
4.71 1.11 0.46
Having failed over the RE now, all that needed is to repeat the same command as before request system software validate add /var/tmp/xxx reboot to install the new firmware on RE0
marquk01@re0-mx480> request system software add /var/tmp/jinstall-14.1R6.4-domestic-signed.tgz reboot
Once the Routing-Engine has upgraded you need to re-enable graceful-switchover and non-routing, first, you will see why in moment.
activate chassis redundancy graceful-switchover
activate routing-options nonstop-routing
After commit synchronising those changes, you will need to set RE0 back to the Master Routing Engine. This will be done by running request chassis routing-engine master switch from RE1 now.
marquk01@re1-mx480>request chassis routing-engine master switch
Toggle mastership between routing engines ? \[yes,no\] (no) yes
Resolving mastership...
Complete. The other routing engine becomes the master.
{backup}
marquk01@re0-mx480>
Now that Graceful Switchover is enable when you run the command you wont see the same warning about traffic being disrupted, this is because Graceful Switchover preserves interface and kernel information allowing the PFE to continue forwarding packets, even though one of the REs is unavailable.
More detail Graceful Switchover can be found on Juniper’s TechLibrary
We will need to save a backup of the currently running and active file system by issuing the command request system snapshot on both primary as well as backup REs
marquk01@RE0-MX480-02> request system snapshot
Doing the initial labeling...
Verifying compatibility of destination media partitions...
Running newfs (899MB) on hard-disk media / partition (ad2s1a)...
Running newfs (100MB) on hard-disk media /config partition (ad2s1e)...
Copying '/dev/ad0s1a' to '/dev/ad2s1a' .. (this may take a few minutes)
Copying '/dev/ad0s1e' to '/dev/ad2s1e' .. (this may take a few minutes)
The following filesystems were archived: / /config
Finally, confirm the code version by running show version invoke-on all-routing-engine. I’ve used the commands show version and show version invoke-on other-routing-engine just because I can put the two outputs into a table-like thing and it looks neater in a post :p
show version RE0⌗
{master}
marquk01@RE0-MX480-02> show version
Hostname: RE0-MX480-02
Model: mx480
Junos: 14.1R6.4
JUNOS Base OS boot [14.1R6.4]
JUNOS Base OS Software Suite [14.1R6.4]
JUNOS Packet Forwarding Engine Support (M/T/EX Common) [14.1R6.4]
JUNOS Packet Forwarding Engine Support (MX Common) [14.1R6.4]
JUNOS platform Software Suite [14.1R6.4]
JUNOS Runtime Software Suite [14.1R6.4]
JUNOS Online Documentation [14.1R6.4]
JUNOS Services AACL Container package [14.1R6.4]
JUNOS AppId Services [14.1R6.4]
JUNOS Services Application Level Gateways [14.1R6.4]
JUNOS Services Captive Portal and Content Delivery Container package [14.1R6.4]
JUNOS Border Gateway Function package [14.1R6.4]
JUNOS Services HTTP Content Management package [14.1R6.4]
JUNOS IDP Services [14.1R6.4]
JUNOS Services LL-PDF Container package [14.1R6.4]
JUNOS Services Jflow Container package [14.1R6.4]
JUNOS Services MobileNext Software package [14.1R6.4]
JUNOS Services Mobile Subscriber Service Container package [14.1R6.4]
JUNOS Services PTSP Container package [14.1R6.4]
JUNOS Services NAT [14.1R6.4]
JUNOS Services RPM [14.1R6.4]
JUNOS Services Stateful Firewall [14.1R6.4]
JUNOS Voice Services Container package [14.1R6.4]
JUNOS Services Crypto [14.1R6.4]
JUNOS Services SSL [14.1R6.4]
JUNOS Services IPSec [14.1R6.4]
JUNOS py-base-i386 [14.1R6.4]
JUNOS Kernel Software Suite [14.1R6.4]
JUNOS Crypto Software Suite [14.1R6.4]
JUNOS Routing Software Suite [14.1R6.4]
show version RE1⌗
{master}
marquk01@RE0-MX480-02> show version invoke-on other-routing-engine
re1:
--------------------------------------------------------------------------
Hostname: RE1-MX480-02
Model: mx480
Junos: 14.1R6.4
JUNOS Base OS boot [14.1R6.4]
JUNOS Base OS Software Suite [14.1R6.4]
JUNOS Packet Forwarding Engine Support (M/T/EX Common) [14.1R6.4]
JUNOS Packet Forwarding Engine Support (MX Common) [14.1R6.4]
JUNOS platform Software Suite [14.1R6.4]
JUNOS Runtime Software Suite [14.1R6.4]
JUNOS Online Documentation [14.1R6.4]
JUNOS Services AACL Container package [14.1R6.4]
JUNOS Services Application Level Gateways [14.1R6.4]
JUNOS AppId Services [14.1R6.4]
JUNOS Services Captive Portal and Content Delivery Container package [14.1R6.4]
JUNOS Border Gateway Function package [14.1R6.4]
JUNOS Services HTTP Content Management package [14.1R6.4]
JUNOS Services Jflow Container package [14.1R6.4]
JUNOS IDP Services [14.1R6.4]
JUNOS Services LL-PDF Container package [14.1R6.4]
JUNOS Services MobileNext Software package [14.1R6.4]
JUNOS Services Mobile Subscriber Service Container package [14.1R6.4]
JUNOS Services NAT [14.1R6.4]
JUNOS Services RPM [14.1R6.4]
JUNOS Services PTSP Container package [14.1R6.4]
JUNOS Services Stateful Firewall [14.1R6.4]
JUNOS Voice Services Container package [14.1R6.4]
JUNOS Services SSL [14.1R6.4]
JUNOS Services Crypto [14.1R6.4]
JUNOS Services IPSec [14.1R6.4]
JUNOS py-base-i386 [14.1R6.4]
JUNOS Kernel Software Suite [14.1R6.4]
JUNOS Crypto Software Suite [14.1R6.4]
JUNOS Routing Software Suite [14.1R6.4]
And with that, we have an upgraded Dual Routing Engine MX Series router! :D Yay! I’ll most likely, now that I’ve got the access, mess about with ISSU upgrade on MX next so keep an eye for that one!