fellas, I have an interesting problem and wonder if I do something evil or just impossible. I want to run a buildah inside a container on mac. for the context it's not just buildah, but a go program that embeds buildah as a module (it shouldn't change much IMO). Im running it on Mac and I constantly get a permission issue error. I tried running it using Colima, using Podman machine, I used

--privileged

flag, I added

CAP_SYS_ADMIN

cap, I tried changing a storage to

vfs

, and turning a rootful mode on Podman

podman machine --set rootful

. if somebody has experienced Im happy to see your input. my next step to try running it in linux just to validate it's pure Mac problem with layer fs or something. but running it on Mac is quite important for dev experience + the company doesn't give linux machines anymore due to security team requirements.

does the container arch differs from mac ? could you share a minimalistic implementation of the issue you are facing, with reproducible steps ? would love to take a look

the program Im running:

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/containers/buildah/define"
	"github.com/containers/buildah/imagebuildah"
	"github.com/containers/storage"
	"github.com/containers/storage/pkg/reexec"
)

func main() {
	reexec.Init()
	buildStoreOptions, err := storage.DefaultStoreOptions()
	ifErr(err)
	buildStore, err := storage.GetStore(buildStoreOptions)
	ifErr(err)

	id, ref, err := imagebuildah.BuildDockerfiles(context.Background(), buildStore, define.BuildOptions{
		Registry:     "jopa",
		Output:       "jopa2",
		Out:          os.Stdout,
		Err:          os.Stderr,
		ReportWriter: os.Stdout,
		IgnoreFile:   "./.dockerignore",
	}, "./Dockerfile")
	ifErr(err)
	fmt.Println(id)
	fmt.Println(ref)
}

func ifErr(err error) {
	if err != nil {
		panic(err)
	}
}

just to be clear

2025-03-08 22:41:50 panic: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted
2025-03-08 22:41:50 
2025-03-08 22:41:50 goroutine 1 [running]:
2025-03-08 22:41:50 main.ifErr(...)
2025-03-08 22:41:50     /go/cmd/tt/t.go:36
2025-03-08 22:41:50 main.main()
2025-03-08 22:41:50     /go/cmd/tt/t.go:19 +0x274

is this the issue you are also facing ?

one of them, but if you run it with

--privileged

it will complain on missing

CAP_SYS_ADMIN

,

could you please explain what it does ? and how it fixes the issue ? now, facing the same issue you mentioned

Writing manifest to image destination
[1/4] STEP 2/8: WORKDIR /app
panic: building at STEP "WORKDIR /app": time="2025-03-08T22:26:54Z" level=warning msg="Failed to read CAP_SYS_ADMIN presence for the current process"
	time="2025-03-08T22:26:54Z" level=warning msg="Failed to read current user namespace mappings"
	panic: kernel does not support overlay fs: unable to create kernel-style whiteout: operation not permitted

	goroutine 1 [running]:
	main.ifErr(...)
		/go/cmd/tt/t.go:36
	main.main()
		/go/cmd/tt/t.go:19 +0x274


goroutine 1 [running]:
main.ifErr(...)
	/go/cmd/tt/t.go:36
main.main()
	/go/cmd/tt/t.go:29 +0x268

you probably need to install

brew install macfuse

in order to support layerfs (I barely understand what it does, now it goes right into my read list). to run it I did

podman run --net=host --security-opt label=disable --security-opt seccomp=unconfined localhost/tt

or

docker

instead of Podman is fine too (podman requires host as a repo for the images, use for docker it's just tt instead of localhost/tt). this go app build an image, I just wanted to learn how can I build a go app that can build the others docker containers, potentially extending it to build packs too. it has only 1 useful line is

id, ref, err := imagebuildah.BuildDockerfiles(context.Background(), buildStore, define.BuildOptions{
		Registry:     "jopa",
		Output:       "jopa2",
		Out:          os.Stdout,
		Err:          os.Stderr,
		ReportWriter: os.Stdout,
		IgnoreFile:   "./.dockerignore",
	}, "./Dockerfile")

simply saying it does

docker ubild

specifying docker ignore and a Dockerfiles, also redirecting stdout to the console. I don't know what most of the fields do, I also need to learn it.

this is a great start, thank you for sharing

Just for the framing… could you elaborate why you’re trying to build a container in a container for devex reasons? And what needs to be build - just Go stuff? I am wondering if this is the easiest path or if there’s an escape hatch.

some of the demo apps don't need a CI pipeline, they just want to connect a

default

branch and say "pls roll it, that much memory and cpu I need". I considered kaniko too, but find it inconvinient. we also need to choose between dockerfile and buildpacks in this scenario

Hmmm… that’s a lot of implementation choices there. Buildpacks would make all the rest not needed. The rest is there as “choices for DX”? Have you tried https://ko.build/ as a lightweight “I just want my Go app inside a container plx” solution?

I saw, it's just for go

True! What else do you need to buld?

usually dockerfiles, but for some services buildpacks would be good, mostly for static sites

I am curious to learn if you found a way to fit all that in a single paradigm @Candyboober. Having that many divergent choices, I’d probably split those and choose an optimized stack depending on the entry choice of what type of deployable is going to be built. Or throw out a little bit of optionality to make things more easy - in the long run also for the devs, as there is less to learn / maintain / onboard to when switching project, etc.

yeah that's fair, I don't see it easy. currently this module in the platform cares only about building given docker images using buildah. probably later it will accept a larger argument to run buildpacks builder. alternatively static sites will also be packed using one more strategy (s3 + cloudflare). all this component most likely will go away as a job with a limited queue and priorities. but it's a long term, currently the goal only to build containers and push them to the given registry, buildah solves it

👌

ok, thanks. either way I will share it on my page and share to you

please share it here, I would also love to have a look