View Issue Details

IDProjectCategoryView StatusLast Update
0015522CentOS-7shimpublic2019-04-04 01:07
Reporterarrfab 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Product Version7.6.1810 
Target Version7.6.1810Fixed in Version 
Summary0015522: Machine with older uefi fw doesn't boot with newer shim (like on Apple 2008 iMac)
DescriptionWith the shim pkgs being rebased on v15 (it was v12 before), there is a new check that is considered critical and then power off the machine instead of loading grub2 (in the normal shim => grub2 => kernel chain)
Steps To Reproduceyum update ; systemctl reboot
iMac on reboot shows :

Failed to set MokListRT: Invalid Parameter
Something as gone seriously wrong: import_mok_state() failed:
 Invalid Parameter

Then machine powers off
TagsNo tags attached.
abrt_hash
URL

Relationships

has duplicate 0015737 closedIssue Tracker Cannot boot ISO using Disk on Key 
related to 0015982 assignedpgreco upgrading to 15-1.el7.centos makes SoftIron OverDrive 1000 unbootable 

Activities

arrfab

arrfab

2018-12-04 07:10

administrator   ~0033203

The problem is that on some older nodes, the uefi implementation in the firmware seems to have a limited in memory size for the efivars.
As since shim v12-2, we enrolled two certificates (previous one and new one) to allow booting on previous grub2/kernel while also permitting booting the new grub2/kernel pkgs, (signed with the new key), the size of efivars grew, but that wasn't considered a critical error by shim on buggy/older firwmare. If secureboot wasn't enabled (the case on such old imac - even on newer ones btw - ) it was still loading grub2 and so machine was booting.

With the rebase to shim v15, that check is now considered critical (as shown in the message with MokListRT) and so shim doesn't allow to boot grub2.

Workaround #1 : on existing machine, before updating, be sure to exclude shim* and mokutil* (so that it can stay at version shim-x64-12-2.el7.x86_64
echo 'exclude=shim*,mokutil*' >> /etc/yum.conf
yum update

Workaround #2 : if you have already updated and also rebooted, you have to use a live media to
- mount /boot/efi
- enter $mount_path to go to /boot/efi/EFI/centos
- cp grubx64.efi over shimx64.efi
- umount
- reboot

We can , as shown in workaround #2, boot directly into grub2 as SecureBoot isn't even possible on the iMac, so bypassing shim completely.
chref

chref

2018-12-04 15:30

reporter   ~0033204

The same happens on Dell PowerEdge R410 with UEFI firmware version 1.14.0 (which is the latest provided by Dell).
arrfab

arrfab

2018-12-04 15:59

administrator   ~0033207

don't know the memory size allowed by Dell for efivars, but you should be able to get back in action with previous shim at least (assuming that you have SecureBoot enabled, in which case you need shim and can't boot straight to grubx64.efi).
eduardok

eduardok

2018-12-04 15:59

reporter   ~0033208

Happens on IBM System x3630 M3, but you can go into Setup, and choose Boot From File, then navigate in the tree to grubx64.efi and boot from it. I'm still to test with latest UEFI firmware.
chref

chref

2018-12-04 16:09

reporter   ~0033209

@arrfab: That's right, downgrading shim-x64 to 12-2 helped.
SecureBoot is disabled here, so, yeah, directly booting grubx64.efi would have worked, too.
benkonrath

benkonrath

2018-12-05 12:40

reporter   ~0033217

I have this error on a MacBook Pro (Retina, 13-inch, Early 2015) after upgrading as well. I dual boot with the original version of OS X (10.10) that came with the system so I guess I have an older uefi firmware. I tried the 'cp grubx64.efi shimx64.efi ' trick but it didn't work, I get the same error message when I try to boot CentOS. I assume this is because secure boot is enabled. Will it work to manually copy files into /boot/efi/EFI/centos from the shim-x64 12-2 package? Is there something else I can try? Thanks in advance for your help.
arrfab

arrfab

2018-12-05 12:52

administrator   ~0033218

@benkonrath : yes, you can extract shimx64.efi from previous shim-x64, or you can also downgrade to shim-x64 (and mokutil) and then exclude shim*,mokutil* from yum

Or you can extract the shimx64.efi from the downloaded pkg, use "rpm2cpio ...|cpio -ivdm" to retrieve it and overwrite the one in /boot/efi/EFI/centos path

Thanks for the report, but one thing that I wasn't aware of was that SecureBoot was a possible thing on Apple hardware, nor then that it was enrolling Microsoft keys by default
Splash

Splash

2018-12-05 14:11

reporter   ~0033222

I have the same issue after upgrading CentOS 7 on a Dell R210 server. BIOS: 1.10.0. Server powers up, runs in to this error when trying to boot UEFI and then powers down.

Capture222.PNG (3,258 bytes)
Capture222.PNG (3,258 bytes)
benkonrath

benkonrath

2018-12-05 14:16

reporter   ~0033223

@arrfab Copying the files from shim-x64 12.-2 to the EFI partition worked. I was able to do it in OS X with the help of rEFInd (used to mount the EFI partition) and The Unarchiver (used to unpack the rpm). Now that I'm back in CentOS, I've downgraded shim-x64 and mokutil and I excluded them from updating.

I'm not actually sure about SecureBoot being enabled on the MacBook Pro. I was only guessing since booting directly with grubx64.efi didn't work.

Thanks for the help on this.
markoh

markoh

2018-12-05 14:27

reporter   ~0033224

I have the same issue on IBM System x3650 M3. Thanks to @eduardok I managed to boot the computer with selecting grubx64.efi.
athompso

athompso

2018-12-05 22:07

reporter   ~0033231

Confirming same issue as Splash (in c33222) but on a Dell R410.
Oddly, one Dell R410 *doesn't* have the issue, while its twin server *does*. We only noticed after swapping the disks in the RAID set between machines due to a hardware failure on the original server.
Nothing worked except copying CENTOS\GRUBX64.EFI over top of EFI\BOOTX64.EFI - still unsure why that is, but that's the step that finally allowed it to boot unattended.
greenpossum

greenpossum

2018-12-06 07:51

reporter   ~0033238

Same thing happened to me today. Upgraded an IBM x3550 M3 from 7.5 to 7.6 online. Didn't come up again. I found the Failed to set MokListRT: Invalid Parameter error on the console. Booting from the installation DVD got the same error. Don't think this server even supports secure boot, couldn't find any mention of it in the BIOS setup, but has UEFI.

In the end I wiped the machine and installed CentOS 6.10. Reason is this is an old unused machine and might be retired any day. The users are much more likely to want a CentOS 6 server if one of the other servers breaks down. If I had found this page earlier I would have tried overwriting the EFI file as suggested above. I'll bookmark this thread for site documentation in the very unlikely case someone wants to run CentOS 7 on it in future.
arrfab

arrfab

2018-12-06 09:36

administrator   ~0033242

So, thanks to @puiterwijk , we have now a patch and also a PR upstream (https://github.com/rhboot/shim/pull/157)
Before we ask Microsoft to sign our newly built shim, we'd like to have some people to test our interim build (the shim, unsigned)
As it doens't affect people with UEFI *without* secureboot, it should fix your issue (it works on my old iMac, the only node on which I can test this ...)

Can you just test this and report feedback please ? (if you still have access to the running node)

rpm -Uvh https://people.centos.org/arrfab/shim/results/shim/20181206092329/15-2.el7.centos.x86_64/shim-unsigned-x64-15-2.el7.centos.x86_64.rpm
cp /usr/share/shim/x64-15-2.el7.centos/shimx64.efi /boot/efi/EFI/centos/

Then reboot and it should boot on shim 15, with the added patch, would still displays the MokListRT issue, but wouldn't consider that a fatal error, and so would still move to grub2 (and so booting normally)

Thanks for your feedback ! the more answers we'll get, the faster we'll submit to microsoft for signing
chref

chref

2018-12-06 09:47

reporter   ~0033243

I can confirm that shim-unsigned-x64-15-2.el7.centos successfully boots on our Dell PowerEdge R410.
knweiss

knweiss

2018-12-06 10:08

reporter   ~0033244

I can confirm that shim-unsigned-x64-15-2.el7.centos.x86_64 successfully boots on a IBM System x3650 M3 with UEFI 1.20 and without Secure Boot after copying test shim manually:
cp /usr/share/shim/x64-15-2.el7.centos/shimx64.efi /boot/efi/EFI/centos/shim.efi (i.e. destination file name was shim.efi not shimx64.efi)

I still see the error message for a very short time but the system continues to load grub and boots fine. Thanks!
vivastar

vivastar

2018-12-06 10:12

reporter   ~0033245

I came into the same issue on a System x3250 M3

I booted as suggested from File then downgraded to those packages
mokutil-12-2.el7.x86_64.rpm shim-unsigned-ia32-12-2.el7.x86_64.rpm shim-unsigned-x64-12-2.el7.x86_64.rpm shim-x64-12-2.el7.x86_64.rpm

Rebooted and all went fine . So I can now just go on holidays and not be called for old servers not booting anymore ...
eduardok

eduardok

2018-12-06 14:24

reporter   ~0033250

How does it choose which EFI image to boot?
I rebooted after running "rpm -Uvh https://people.centos.org/arrfab/shim/results/shim/20181206092329/15-2.el7.centos.x86_64/shim-unsigned-x64-15-2.el7.centos.x86_64.rpm ; cp /usr/share/shim/x64-15-2.el7.centos/shimx64.efi /boot/efi/EFI/centos/" yet it powered off (booted the wrong efi instead of shimx64.efi).
I had to manually boot from file shimx64.efi, which indeed printed the message like you said, but at least booted.
knweiss

knweiss

2018-12-06 14:35

reporter   ~0033251

@eduardok The command "efibootmgr -v" will show you the path/filename for each EFI boot target.
arrfab

arrfab

2018-12-06 14:35

administrator   ~0033252

@eduardok : basically if you overwrite shimx64.efi with the cp command, it's the default entry for centos install, so it should reboot on that one.
The fact that you selected it directly as boot entry and that it worked is also what I wanted to get as feedback
oranges2apples

oranges2apples

2018-12-06 22:01

reporter   ~0033255

MacMini here, disk encrypted, replacing shimx64.efi with the one from the shim-unsigned-x64-15-2.el7.centos.x86_64.rpm fixed the issue.
Same issue after a yum update happened here and being a bit more new to CentOS took me a little bit of extra brain power to figure out as I couldn't find /boot/efi/ folder at all on my machine.
Which Im sure might be obvious to most here, it wasn't as obvious for me, and so I only provide the following in the event that someone else out there might be as clueless as me.
Booted off a CentOS USB Installer I made when I first setup the system and chose the Troubleshooting menu option and then the Rescue option.
From there I said Continue which then asked me for my HDD password, and mounted it under /mnt/sysimage.
I chrooted that directory as the screen gave direction for and then I was able to get to /boot/efi/EFI/centos/ as advised above.
Unfortunately, I couldn't just download the rpm file posted above as I don't think the computer had DNS or Internet setup on it while in this recovery mode, so I downloaded the file on my personal computer and stuck it onto a USB drive and mounted that drive on the centos system (mount -t vfat /dev/sdc1 /mnt/usbdisk/)
I copied that file to the CentOS system and then ran rpm -Uvh /home/user/shim-unsigned-x64-15-2.el7.centos.x86_64.rpm
then did cp /usr/share/shim/x64-15-2.el7.centos/shimx64.efi /boot/efi/EFI/centos/
typed exit to get out of the chroot
then shutdown -r now
the restart came up with a lot of stuff on the screen, errors and warnings and then restarted again all on its own.
The 2nd restart it booted up just fine without any issue at all.
kenhom

kenhom

2018-12-07 00:25

reporter   ~0033256

I can confirm that shim-unsigned-x64-15-2.el7.centos successfully boots on our Dell PowerEdge R410
thanks!
rhbauman

rhbauman

2018-12-07 14:59

reporter   ~0033260

I can confirm that shimx64.efi from shim-unsigned-x64-15-2.el7.centos successfully boots on our Dell PowerEdge R310. Secure boot is not available on this server. Thanks for the quick fix.
arrfab

arrfab

2018-12-07 17:30

administrator   ~0033264

Thanks for the feedback.
I launched a CI validation test against that build to test Secureboot validation : https://ci.centos.org/job/CentOS_7_SecureBoot_validation_SelfSigned/20/console

It's now waiting for Review and then Microsoft can sign the shim again and we can then sign/push to updates
https://github.com/rhboot/shim-review/issues/45
jb_alvarado

jb_alvarado

2018-12-08 20:54

reporter   ~0033276

Hi,
I only wanted to report that I have here 2 identical machines (x3650 M3). On both I apply shim-unsigned-x64-15-2.el7.centos. One starts now fine but the second one hangs on exactly that screen with the message:

Failed to set MokListRT: Invalid Parameter
Something as gone seriously wrong: import_mok_state() failed:

Did you have any idea why this is happen?

Selecting grubx64.efi works...
arrfab

arrfab

2018-12-09 08:26

administrator   ~0033277

@jb_alvarado : if you're sure that you have indeed copied/overwritten the shimx64.efi on that system with the one from shim-unsigned pkg, it should boot.
Can you ensure that such node is configured to boot on that file ?
efibootmgr -v

And then sha256sum /boot/efi/EFI/centos/shimx64.efi
jb_alvarado

jb_alvarado

2018-12-09 09:15

reporter   ~0033278

@arrfab ,
here are the outputs:

efibootmgr -v:

BootCurrent: 003D
Timeout: 10 seconds
BootOrder: 0010,000E,000F,000C,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000A,000B,000D
Boot0000* CD/DVD Rom VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0c)
Boot0001* Floppy Disk VenMedia(0c588db8-6af4-11dd-a992-00197d890238,00)
Boot0002* Hard Disk 0 VenMedia(0c588db8-6af4-11dd-a992-00197d890238,08)
Boot0003* PXE Network VenMedia(0c588db8-6af4-11dd-a992-00197d890238,06)
Boot0004* Hard Disk 1 VenMedia(0c588db8-6af4-11dd-a992-00197d890238,09)
Boot0005* Hard Disk 2 VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0a)
Boot0006* Hard Disk 3 VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0b)
Boot0007* Hard Disk 4 VenMedia(0c588db8-6af4-11dd-a992-00197d890238,0e)
Boot0008* USB Storage /Pci(0x1a,0x7)/USB(0,0)/HD(1,MBR,0x985b0db,0x1b0,0x4638)
Boot0009* Diagnostics VenMedia(0c588db8-6af4-11dd-a992-00197d890238,da)
Boot000A* iSCSI VenMedia(0c588db8-6af4-11dd-a992-00197d890238,04)
Boot000B* iSCSI Critical VenMedia(0c588db8-6af4-11dd-a992-00197d890238,05)
Boot000C* Legacy Only VenMedia(0c588db8-6af4-11dd-a992-00197d890238,ee)
Boot000D* Embedded Hypervisor VenMedia(0c588db8-6af4-11dd-a992-00197d890238,01)
Boot000E* oVirt Node Next HD(1,GPT,a3a0d01f-2c82-4fbf-92b0-73eecffe85bb,0x800,0x64000)/File(\EFI\centos\shimx64.efi)
Boot000F* CentOS Linux HD(1,GPT,85b8cd7c-4c1c-4540-9e5b-6d25b3af0f47,0x800,0x64000)/File(\EFI\centos\shim.efi)
Boot0010* CentOS HD(1,GPT,f840a249-cba6-402a-aa50-0b2a21b8cc93,0x800,0x64000)/File(\EFI\centos\shimx64.efi)

sha265sum:
47391a4b25918b7f414420d8f651e302b287a96e0747019d76f705fbf361974e /boot/efi/EFI/centos/shimx64.efi

Before I apply this update, the system was immediately shutdown, not hanging on that screen.
greenpossum

greenpossum

2018-12-10 10:26

reporter   ~0033294

@oranges2apples
>Unfortunately, I couldn't just download the rpm file posted above as I don't think the computer had DNS or Internet setup on it while in this recovery mode

Actually you can activate the network interface manually. For future reference, it goes like this:

ip addr
# locate the appropriate interface
ifconfig eth0 192.168.20.20 gw 192.168.20.1 # for example

Unfortunately you cannot start a ssh service on it, but you can sftp out, using IP addresses, and fetch the RPM you need from another local machine

sftp 192.186.20.2
etc
appleII

appleII

2018-12-20 01:23

reporter   ~0033391

Same issue on my 2008 mac mini. Verifying the suggested unsigned package w/ shimx64.efi fixed the issue.
@arrfab - the issue summary was terrific for google to help me find this. Also, great job on the clarity of your answers for this issue.
@oranges2apples post gave me hope in resolving this issue. Thanks!
@greenpossum and @oranges2apples , getting the apple mac networking running from the 'rescue shell' looks harder than we might think. The ip addr command only listed the loopback interface, no other interfaces. Also, ip link command did not show other devices. The lshw -c network did show an "UNCLAIMED" for the MCP79 Ethernet from Nvidia Corp, and I stopped here because I didn't want to chase driver or kmod details.
dawgly

dawgly

2018-12-21 01:44

reporter   ~0033412

iMac 2008 (1,7)

After obtaining the unsigned shim, replacing the existing file with same name, configuring yum to ignore shim and mokutil, conflict arises during yum update:
"Processing Conflict: kernel-3.10.0-957.1.3.el7.x86_64 conflicts shim-x64 < 12-2
[...]
Error: kernel conflicts with shim-x64-12-1.el7.centos.x86_64"

Still familiarizing myself with the moving parts and will report back with any good news.
arrfab

arrfab

2018-12-21 12:47

administrator   ~0033420

@dawgly : well, we're still waiting on Microsoft to review the new shim build and nothing we can do for now :(
But if you haven't removed shim and just install shim-unsigned and that you just overwrote the correct files, yum shouldn't complain for now.
In case you have completely removed shim-x64, reinstall it, and copy/overwrite gain shimx64.efi like described in the workaround mentioned above
TrevorH

TrevorH

2018-12-21 13:04

manager   ~0033423

You also need the newer shim 15.x package to install kernels from 7.6 as yum tells you. You will need to update to the newer shim package and then replace the file you want to replace.
bako

bako

2019-01-13 01:26

reporter   ~0033570

@arrfab Why do you have to wait for Microsoft? Can't you just update the repo? I'm using it on a FreeNAS BHYVE so I don't need the Microsoft's approval.
arrfab

arrfab

2019-01-13 07:38

administrator   ~0033573

@bako : because we built the shim-unsigned pkg (not signed by Microsoft) and build the proper shim needs the shimx64.efi signed back , so we can't push an updated shim pkg (shim-x64) until we have it back. As simple as that (as all x64 nodes we know so far only trust the Microsoft key by default in MoK, reason why we need our shim to be signed : to be able to load properly)
emodeca

emodeca

2019-01-21 19:54

reporter   ~0033650

Had this same issue on an R410. For anyone else with an affected Dell Server, as a workaround, you can change add a UEFI boot entry to point to grubx64.efi directly without having to boot to a rescue OS. F11 > UEFI Boot Manager > Add boot entry > etc ...
To prevent it from shutting itself off every time it reboots, remember to move the new entry to the top and/or remove the old default shimx64.efi entry.

Thank you for the info and your assistance with this ticket. Saved me having to find and send a contractor onsite to smash this box with a hammer. Much appreciated!
arrfab

arrfab

2019-01-28 15:35

administrator   ~0033713

Just to add that some aarch64 boxes are also affected by this : just had myself that issue on ThunderX, so same workaround described above works, but we should have a faster rebuild for aarch64 as we don't depend on Microsoft for that architecture (MS keys aren't rolled in aarch64 MOK)
brucequinton

brucequinton

2019-02-04 14:07

reporter   ~0033771

All, I just had this issue with an IBM x3550 m2. Copying the \EFI\centos\grubx64.efi over the \EFI\centos\shimx64.efi did not work for me because the UEFI Boot Manager entry was pointed to \EFI\centos\shim.efi.
I overwrote \EFI\centos\shim.efi with \EFI\centos\grubx64.efi (i.e. cp \EFI\centos\grubx64.efi \EFI\centos\shim.efi ), and was able to boot.
mattpie

mattpie

2019-02-12 20:50

reporter   ~0033819

I ended up using efibootmgr to create a new boot item to save my worry that a future update would hose my system. Hardware is a Mac MIni (2011-ish), so I don't need signed boot. I ran this after doing a yum update but before rebooting. I assume it can be done from a Live media/rescue boot.

Create a new boot entry pointing to grubx64.efi:
efibootmgr -c -L "CentOS-grubx64" -L \\EFI\\centos\\grubx64.efi

Then check "efibootmgr -v" to check the boot order and the entries and make sure the grubx64 one is before the original (Centos). On reboot, system went straight into GRUB menu and booted.
arrfab

arrfab

2019-02-22 10:55

administrator   ~0033885

We finally got the shim signed back from Microsoft so I built the new rpm for it.
It was validated by the following CI job :
https://ci.centos.org/job/CentOS_7_SecureBoot_validation_SelfSigned/21/console

The new rpm packages (not rpm-signed yet) are available here :
https://people.centos.org/arrfab/shim/results/shim-signed/20190222095040/15-2.el7.centos.x86_64/

If you want to test, it should a simple as :
cd /tmp
 wget https://people.centos.org/arrfab/shim/results/shim-signed/20190222095040/15-2.el7.centos.x86_64/shim-x64-15-2.el7.centos.x86_64.rpm https://people.centos.org/arrfab/shim/results/shim-signed/20190222095040/15-2.el7.centos.x86_64/mokutil-15-2.el7.centos.x86_64.rpm
 rpm -Uvh mokutil-15-2.el7.centos.x86_64.rpm shim-x64-15-2.el7.centos.x86_64.rpm

And then "systemctl reboot" (obviously)
Feedback would be appreciated and also the plan would be to only rpm-sign and push those pkgs to [updates] repo next monday.
dawgly

dawgly

2019-02-24 22:48

reporter   ~0033895

iMac (2008) 7,1
CentOS 7.5 GNOME Desktop installed via USB

Followed the test instructions recently mentioned above:
- obtained two files using wget
- rpm'ed the two files
- systemctl rebooted

Next:
yum updated
rebooted after update completed

logged in = success! Thank you!!

Does this mean that Centos 7.7 might be plug-and-play for these older machines?
arrfab

arrfab

2019-02-25 07:06

administrator   ~0033896

@dawgly : hopefully yes, but my plan would be to have this rolled-in in a respin iso image too (we do that on monthly basis, so the one from february would be 7.6.1810 + updates - including shim and so would be installable without any issue.
I'll ping Johnny about this (producing the monthly iso images)
markoh

markoh

2019-02-25 07:48

reporter   ~0033897

Hi,

I can confirm that the update works fine on IBM x3650 M3 servers.

Thanks!
arrfab

arrfab

2019-02-26 06:23

administrator   ~0033904

Closing as the following packages were signed/pushed to [updates] repo :

mokutil.x86_64 15-2.el7.centos updates
shim-ia32.x86_64 15-2.el7.centos updates
shim-unsigned-ia32.x86_64 15-2.el7.centos updates
shim-unsigned-x64.x86_64 15-2.el7.centos updates
shim-x64.x86_64 15-2.el7.centos updates

Issue History

Date Modified Username Field Change
2018-12-04 06:58 arrfab New Issue
2018-12-04 07:10 arrfab Note Added: 0033203
2018-12-04 07:10 arrfab Status new => confirmed
2018-12-04 15:30 chref Note Added: 0033204
2018-12-04 15:59 arrfab Note Added: 0033207
2018-12-04 15:59 eduardok Note Added: 0033208
2018-12-04 16:09 chref Note Added: 0033209
2018-12-05 12:40 benkonrath Note Added: 0033217
2018-12-05 12:52 arrfab Note Added: 0033218
2018-12-05 14:11 Splash File Added: Capture222.PNG
2018-12-05 14:11 Splash Note Added: 0033222
2018-12-05 14:16 benkonrath Note Added: 0033223
2018-12-05 14:27 markoh Note Added: 0033224
2018-12-05 22:07 athompso Note Added: 0033231
2018-12-06 07:51 greenpossum Note Added: 0033238
2018-12-06 09:36 arrfab Note Added: 0033242
2018-12-06 09:36 arrfab Status confirmed => feedback
2018-12-06 09:47 chref Note Added: 0033243
2018-12-06 10:08 knweiss Note Added: 0033244
2018-12-06 10:12 vivastar Note Added: 0033245
2018-12-06 14:24 eduardok Note Added: 0033250
2018-12-06 14:35 knweiss Note Added: 0033251
2018-12-06 14:35 arrfab Note Added: 0033252
2018-12-06 22:01 oranges2apples Note Added: 0033255
2018-12-07 00:25 kenhom Note Added: 0033256
2018-12-07 14:59 rhbauman Note Added: 0033260
2018-12-07 17:30 arrfab Note Added: 0033264
2018-12-08 20:54 jb_alvarado Note Added: 0033276
2018-12-09 08:26 arrfab Note Added: 0033277
2018-12-09 09:15 jb_alvarado Note Added: 0033278
2018-12-10 10:26 greenpossum Note Added: 0033294
2018-12-20 01:23 appleII Note Added: 0033391
2018-12-21 01:44 dawgly Note Added: 0033412
2018-12-21 12:47 arrfab Note Added: 0033420
2018-12-21 13:04 TrevorH Note Added: 0033423
2019-01-13 01:26 bako Note Added: 0033570
2019-01-13 07:38 arrfab Note Added: 0033573
2019-01-21 19:54 emodeca Note Added: 0033650
2019-01-23 14:21 TrevorH Relationship added has duplicate 0015737
2019-01-28 15:35 arrfab Note Added: 0033713
2019-02-04 14:07 brucequinton Note Added: 0033771
2019-02-12 20:50 mattpie Note Added: 0033819
2019-02-22 10:55 arrfab Note Added: 0033885
2019-02-24 22:48 dawgly Note Added: 0033895
2019-02-25 07:06 arrfab Note Added: 0033896
2019-02-25 07:48 markoh Note Added: 0033897
2019-02-26 06:23 arrfab Status feedback => resolved
2019-02-26 06:23 arrfab Resolution open => fixed
2019-02-26 06:23 arrfab Note Added: 0033904
2019-04-04 01:07 pgreco Relationship added related to 0015982