View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0006228||CentOS-6||Cloud-Images||public||2013-01-30 16:40||2016-12-21 15:26|
|Platform||Cloud AWS, Amazon EC2||OS||OS Version|
|Target Version||Fixed in Version|
|Summary||0006228: Official CentOS 6 Amazon EC2 images backed by instance store seem not available|
|Description||I'm referring to your Cloud AWS wiki page here:|
We have been very happy to see that there are now official CentOS AMIs on the AWS Marketplace. But unfortunately, I can't find any of the instance-store/s3 backed AMIs that are listed on your wiki on Marketplace. The AMI IDs given on the wiki don't seem to not exist.
So we tried to quickly convert your EBS-backed AMI (this one does exist!) into an instance-store backed one but the process fails apparently because of license limitations and "product code" related issues likely linked to the distribution through AWS Marketplace. See here:
So it would be fantastic if you could make the instance-store backed AMIs as are listed on your wiki page also actually available in Marketplace as this is the kind of AMI we are looking for.
Further, I think it would be great if you could make the AMIs also available for download from outside Marketplace. It seems the Marketplace distribution can add unnecessary hassle (having to subscribe to your product first, accepting the EULA, then running into issues with the attached product code, etc.).
We'll be very grateful for any help you can provide to fix our problem.
|Tags||No tags attached.|
I was about to open up a bug re: Marketplace product codes and restrictions making the AMIs dangerous to use in production.
Background: I have been working heavily with RHEL AMIs in AWS - well over 100 instances in dev and production right now.
The problem with Marketplace AMIs is they prevent you from mounting any root image as a secondary device. So if you have a hiccup, the recovery option of stopping the instance, mounting the root volume on another instance, fixing the file system/config file/Puppet/Chef config is gone. As DevOps, I cannot give up that useful tool/strategy in exchange for using the blessed CentOS instances.
I myself needed an HVM AMI of CentOS (building a big Riak cluster) and was trying to convert the existing Marketplace PV AMI to HVM. It would be nice if the Official CentOS AMIs were in the Community AMI pool. Or replicated into both Marketplace and Community. Yes, Community can be a bit of a cesspool searching for images, but having AMI numbers on the CentOS wiki for official AMI instances in community (similar to checking your MD5 sums on tarballs), should help with that.
LKKM, I've looked at the AMIs Sam Bashton has been building in the community. His stuff looks pretty good - see http://www.bashton.com/blog/2013/centos-6-4-ami-available/. My group is currently working through a decision to base off of his AMIs (and he has HVM), create our own CentOS base AMIs from scratch (Sam posts all of his tools), or - and unfortunately this is the leader with Management - go with AWS Linux. My personal preference is something with the CentOS.org stamp-of-approval but Marketplace restrictions make the current AMIs unsuitable.
If I am getting this right, you would like to have:
1) - Marketplace AMIs (xen and hvm ones)
2)- same version as in 1) but in the Community pool (can't be the same AMI, due to AWS restrictions if I understand you correctly -> any pointer on the limitations?)
3) - gather good wills from bashton.com and you? and avoid duplicating efforts
Ideally, what I would like to have are Marketplace AMIs xen and hvm, without a 'product code' marking as the product code prevents marking it prevents mounting an EBS root volume as a secondary device on another instance. This last part is an operations necessity on my part.
LKKM would like Instance backed CentOS AMI, but Amazon only allows EBS-backed AMIs in the Marketplace, so they would need to go through the Community or roll-their-own.
My idea was that if CentOS.org could also establish its AWS Account ID with "blessed" AMIs in the Community side of things, these two tasks could be accomplished.
re: #3, I cannot speak for Mr. Bashton, but yes, it would avoid my duplicating efforts or possibly having to use a distro other than CentOS for my project.
I was very happy to find CentOS in the Marketplace and had already started implementing based on those AMIs until I ran into my two show-stoppers, which appear to be caused by being in the Marketplace. That said, I noticed today that the Basho Riak AMIs (AWS Linux based) do not have their root volumes tagged with a Product Code. I do not know if that is because of it being AWS Linux based, an option when registering with the Marketplace, or some other item.
So in summary, what I really want is to be able to mount the EBS root volumes from instances from a blessed AMI as a secondary device.
we do have an AWS account - but its backed by my personal Credit Card - and am reluctant to have an image/instance posted from there that is then usable by the entire world.
However, I am willing to push AMI's backing stores along with the script required to instantiate it using your own account. This would solve the issue that LKKM has, in that he wants an instance backed image. And it would also solve the issue that ksquared has, in that he wants an official CentOS image to consume. However, it comes with an additional step of having to upload an image to start the process, rather than just consuming already established.
Is that a reasonable middle ground?
That sound great to me. As it turns out, our project turned from Riak to DynamoDB so my pressing need for HVM-sized instances went away. But having AMIs that I can mount the roots as secondary drives, that I can trace the provenance back to CentOS (and esp. to you good sir!), and that I can use to generate Golden AMIs for deployments under our production account is something I believe would be extremely useful to me and other folks.
|Sounds great to me as well! We are further customizing the AMI and uploading it to our account already anyway. So it's fine if the instance-store backed AMI is distributed as not "directly consumeable". All we'd want is having an AMI from "official" source that we can then use as the starting point.|
|I am using the "CentOS 6.5 (x86_64) - Release Media" (ami-b6bdde86) image and ran into the issue of not being able to mount a volume as a secondary drive due to the marketplace code restrictions. I'm not sure why the official CentOS images use a marketplace code but it should be removed so we can perform boot and login recovery.|
|My organization really wants to use instance-store CentOS images but we don't want to do this from scratch. Is the EBS backed image creation process documented/in source somewhere? Taking an EBS backed image and going to instance store is not trivial.|
|We'd love to have HVM instances, we currently aren't able to utilize the new r3 instance types in AWS due to the lack of HVM support.|
The restrictions on the use of Marketplace AMIs makes centos unusable for me on AWS.
The primary way I debug boot problems or fix corrupted root partitions is to remount the /dev/sda from the problematic instance onto another image as an extra mount point (/dev/sdf for instance) and then inspect / fix the partition.
But with these ridiculous Marketplace restrictions I can't do this fundamental set of operations.
I can't find any trustworthy centos6.5 AMIs that aren't in the Marketplace. I don't have this problem with Ubuntu images.
Is there a known workaround? Is there an alternative location to get non-marketplace official centos images?
There is a work-around. The following was suggested by AWS support and I was able to successfully make an image that no longer has the Marketplace restrictions on the root volume.
"Using a newly-created EBS volume, make a block copy of the Marketplace instance's root volume. Attach the new volume to an instance, and image it."
I created an EBS volume of the same size as the existing CentOS root volume, added it to an existing CentOS AMI, then used dd to do the block copy. Then I was able to add that EBS volume as a root volume to a new instance and create an image from it.
I can now use that newly created image to deploy CentOS AMI's with unrestricted volumes as needed. So now if something in the boot sequence gets messed up (bad kernel, etc), I'm able to attach that volume to another instance and fix it.
Hope this helps.
Here are the specific directions the AWS support engineer gave me:
1. Launched an instance from ami-b6bdde86 (CentOS 6.5 x86_64 us-west-2, as listed on https://aws.amazon.com/marketplace/pp/B00IOYDTV6).
2. Created an 8G EBS volume (no snapshot).
3. Attached the secondary volume to my instance at /dev/sdf (internally, showed up as /dev/xvdj).
4. Formatted the volume: mkfs -t ext4 /dev/xvdj
5. Copied the root volume using dd: dd bs=65536 if=/dev/xvde of=/dev/xvdj
6. Stopped the instance, swapped the volumes around, and brought it up again.
During the brief period I spent testing, I didn't notice any adverse effects, and I was able to freely mount my new root volume as whichever device I pleased.
jake1138: Thanks! We'll see if that works for our use case.
Appreciate the detailed instructions.
I presume you don't really have to do the mkfs step since dd is a raw copy and I believe it will recreate its own filesystem.
I also presume that once you swaped the volumes around and were running on your Marketplace restrictionless volume, you can create a new AMI from the instance and the create derived ami's that also don't have Marketplace restrictions?
That's funny that you mention the mkfs step being unnecessary because I also thought that as well and skipped that step and the volume wouldn't mount or something, can't remember now. I can't explain why mkfs makes it work. All I know is that on the second try, I did the mkfs first then dd and it worked.
Yes, your new AMI and derived AMI's will no longer have the Marketplace restriction as they use a volume with no product code. The same will be true for any snapshots you make.
We've got a pressing need for HVM instances per nbartolemeo, and (formerly) KSquared. The problem is exacerbated by the increasing unavailability of m1.small instances in US East, and the need to migrate to t2.micro, etc. which requires HVM.
No immediate need for instance store.
In the near future we also want to migrate to CentOS 7 and am hearing there may no longer be AMIs for that.
Could you possibly:
* Discuss possibilities for HVM and CentOS 6.5 or 7 and possible workarounds
* Discuss current thinking on AWS and CentOS 7
The workaround for Marketplace looks great ... thanks ... we'll look into it!
Instructions for removing marketplace restriction for CentOS 7:
1. Launch an instance from the marketplace.
2. Create a new 30GB EBS volume (no snapshot).
3. Attach the new volume to the instance at /dev/sdf (internally /dev/xvdf).
4. Format the volume: mkfs -t xfs /dev/xvdf
5. Copy the root volume using dd: dd bs=65536 if=/dev/xvda of=/dev/xvdf
6. Stop the instance.
7. Detach the new volume and attach it to the instance at /dev/sda1
8. Start the instance.
|I forgot to mention that I created a 30GB EBS volume on my original CentOS 7 instance so that is why I created a new 30GB EBS volume. Make sure whatever size you originally had that you create a new volume of the same size in step 2. To some that may be obvious but to others it may not. Good luck!|
dd'ing a live, mounted file system is always dangerous. It may work, but you don't know what you might have caught in mid-modification.
I'm with "robinpc" now, looking for an HVM for CentOS 6. Changes Amazon made in Amazon Linux 15.3 hosed our CentOS 6/Amazon Linux dual distro development environment (namely, making default python now be 2.7, which moves site-packages compared to CentOS 6).
I wouldn't call it dangerous if the source is a new instance with nothing running on it and the target is a new volume. If you have a better solution, feel free to share.
As for HVM for CentOS 6, perhaps this guy's blog post will help:
It is dangerous - always has been. There is something running on it - the OS. Block I/O might be small, but it is not zero.
This is not an image that can be trusted in a Production environment. Yes, it will likely work, but it's not a provably clean installation.
I don't have a "better" solution, which is why I added my metoo to the bug 2 years ago.
That is true. You really can't trust a file system in a production environment that was created in this way. But there is a solution to that.
Amazon now allows you to mount a root volume to another AMI even with the marketplace restriction. I was able to mount a root volume from one CentOS 7 image to another CentOS 7 system and access the filesystem. So that is great news for those of us who want to be able to do boot repair and file recovery.
The only gotcha is that the CentOS people failed to generate a unique UUID for each filesystem created so all the images you create from their AMI has a filesystem with the same UUID. By default, you can't mount an XFS filesystem that has a duplicate UUID. But you can if you use the "-o noouid" option, like so:
mount -o nouuid /dev/xvdf1 /mnt/disk/
So for me, this is sufficient. I no longer have a need to remove the marketplace restriction. But you could do it safely now if you wanted to. Just attach the source root volume and a destination root volume but do not mount either and use mkfs and dd to clone one to the other.
|oh wow, that's a change. You couldn't do that 2 years ago. I wonder if that's new with the CentOS 7 AMI. Thanks for the heads up, I'll go try.|
As for those of you asking about HVM for CentOS 6, I see they have this image on the marketplace:
CentOS 6 (x86_64) - with Updates HVM
But I don't see HVM for CentOS 6.5.
And for those that aren't aware, the CentoS 7 AMI uses HVM.
CentOS 7 (x86_64) with Updates HVM
|This is still an issue for those who want to create instance-store Centos7 AMIs. Would be create if there was an official instance-store AMI for Centos7 that did not have issues with product codes.|
so still no resolution for this one? i'm trying to build one from scratch and i get this error:
xlblk_init: register_blkdev major: 202
blkfront: xvde1: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled;
blkfront: xvdf: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled;
xvdf: unknown partition table
dracut Warning: No root device "block:/dev/disk/by-label/\x2f" found
dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.
dracut Warning: Signal caught!
dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-642.1.1.el6.x86_64 #1
I've been happily building custom images off the official CentOS 7 marketplace AMI, but recently wanted to use an instance type that's not on the list at https://aws.amazon.com/marketplace/pp/B00O7WM7QW (namely, x1.16xlarge), which AWS does not permit, due to the product code.
Jake's thread from 2015-06-25 above mostly worked, except for data corruption — the danger of dd'ing a live, mounted file system that ksquared mentioned. In particular, tons of "Structure needs cleaning" errors once I launch a new instance with the new AMI. I guess I just got unlucky? Twice?
My version of the process, which thus far is working:
1. Start a new instance with the desired CentOS AMI with DeleteOnTermination=false, and then terminate it (all you want is the volume)
2. Start another host instance, or just have another EC2 instance in the same AZ, and stop it
3. Attach the volume from #1 to the instance from #2, as /dev/sdc
4. Create a new blank volume and attach it to the instance from #2, as /dev/sdf
5. Start the instance from #2, and mkfs + dd as in Jake's instructions — crucially, now, neither volume that dd is reading/writing is mounted
6. Stop the instance, detach all the volumes
7. (Re-)Attach the volume from #4 as /dev/sda1
8. Create image from the instance
Many thanks to Jake and the original AWS support person he talked to for figuring out the basic process.
|I concur, that is the correct procedure now. It is what I tried and failed to do 3+ years ago as Step #3 was blocked by the product code back then. Glad that changed!|
|2013-01-30 16:40||lkkm||New Issue|
|2013-05-26 03:48||ksquared||Note Added: 0017506|
|2013-05-26 16:28||toracat||Status||new => assigned|
|2013-05-27 07:30||tru||Note Added: 0017508|
|2013-05-29 17:14||ksquared||Note Added: 0017518|
|2014-01-05 15:email@example.com||Category||-OTHER => Cloud-Images|
|2014-01-05 15:firstname.lastname@example.org||Note Added: 0018786|
|2014-01-05 15:email@example.com||Status||assigned => feedback|
|2014-01-05 20:45||ksquared||Note Added: 0018792|
|2014-01-16 09:59||lkkm||Note Added: 0019067|
|2014-01-16 09:59||lkkm||Status||feedback => assigned|
|2014-03-24 22:55||jake1138||Note Added: 0019541|
|2014-05-22 18:11||nsja||Note Added: 0019780|
|2014-06-05 18:27||nbartolomeo||Note Added: 0019814|
|2014-07-24 21:27||rberger||Note Added: 0020522|
|2014-07-24 21:48||jake1138||Note Added: 0020524|
|2014-07-24 22:07||jake1138||Note Added: 0020525|
|2014-07-24 22:15||rberger||Note Added: 0020526|
|2014-07-24 22:41||jake1138||Note Added: 0020527|
|2014-09-29 18:51||robinpc||Note Added: 0021022|
|2015-06-25 16:49||jake1138||Note Added: 0023491|
|2015-06-25 16:56||jake1138||Note Added: 0023492|
|2015-06-25 17:01||ksquared||Note Added: 0023493|
|2015-06-25 17:25||jake1138||Note Added: 0023494|
|2015-06-25 17:58||ksquared||Note Added: 0023495|
|2015-06-26 14:59||jake1138||Note Added: 0023503|
|2015-06-26 15:05||ksquared||Note Added: 0023504|
|2015-06-26 15:16||jake1138||Note Added: 0023505|
|2015-06-26 16:22||jake1138||Note Added: 0023507|
|2015-11-11 00:14||davidpoxon||Note Added: 0024818|
|2016-06-05 15:16||cristian.leonte||Note Added: 0026787|
|2016-12-21 05:31||chrisbrown||Note Added: 0028225|
|2016-12-21 15:26||ksquared||Note Added: 0028227|